Knowledge Base

  1. Home
  2. Knowledge Base
  3. Technical Support
  4. Script Commands
  5. rtcSetTestExpectation()

rtcSetTestExpectation()

Indicate expected outcome for the specific probe execution. This is used to decide if a test has successfully passed or failed.

The condition is evaluated at the end of the test, based on collected information and metrics. The command can be placed anywhere in the script and can appear multiple times with different constraint values.

Arguments:

NameTypeDescription
criteriastringThe criteria to test. See below for the available options
start-eventstringOptional. If not given, defaults to “start”
The starting point in time for the evaluation. The values that can be given here are either “start” or events provided by the use of rtcEvent()
end-eventstringOptional. If not given, defaults to “end”
The ending point in time for the evaluation. The values that can be given here are either “end” or events provided by the use of rtcEvent()
messagestringMessage to invoke if criteria isn’t met
levelstringLevel of expectation:

 

  • “error” – error occured – fail the test
  • “warning” – consider this as a warning
  • Default value: error

Criteria

A criteria is comprised of the metric to test, an operator and a value.

For example: “video.in > 0” will evaluate that the number of incoming video channels is greater than 0.

Operators

The available operators for the criteria are:

  • ==
  • >
  • <
  • >=
  • <=
  • !=

Criteria metrics

The criteria is defined as a chained definition of the object we wish to evaluate.

The chain hierarchy is:

Media type (audio / video) . Direction (in / out) . channels . Evaluated criteria

Where the channels level is optional, i.e. both formats will work:

  1. video.in.bitrate – Evaluates the total bitrate of all incoming video channels
  2. audio.out.channel.bitrate – Evaluates the bitrate of each outgoing audio channel separately
Evaluated criteriaMedia typeDirectionGlobal (all channels)Per channelDescription and example
Connection durationEvaluate the duration of the whole peer connection session in seconds
“connectionDuration > 60”
Number of channelsaudio | videoin | outYesNoEvaluate the number of channels
“video.out >= 1”
Channel is emptyaudio | videoin | outYesYesEvaluate if the total number of sent/received packets is zero
“video.in.empty == 0”
Bitrateaudio | videoin | out Yes Yes“audio.out.bitrate >= 30” – evaluate the TOTAL aggregated bitrate of all outgoing audio channels
“audio.out.channel.bitrate > 30” – evaluate the value of EACH outgoing audio channel’ bitrate (evaluating all channels, per channel)

 

Also supports min/max postfix: “video.out.bitrate.max < 500” or “audio.in.bitrate.min > 30”

Packet lossaudio | videoin | outYesYesEvaluate channels’ packet loss (in %)
“audio.in.packetloss < 2”
Packets lostaudio | videoin | outYesYesEvaluate the number of packets lost on the channel
“audio.in.packetlost < 20”
Dataaudio | videoin | outYesYesEvaluate total amount of sent/received bytes in the whole session
Packetsaudio | videoin | outYesYesEvaluate the total number of packets sent/received
Roundtripaudio | video outYesYesEvaluate the average round trip time”audio.out.channel.roundtrip < 100″

 

Also supports min/max postfix: “video.out.roundtrip.max < 200”

Jitteraudio | videoinYesYesEvaluate the average jitter value on the network (based on jitterReceived getstats metric)
“audio.in.jitter < 200”

 

Also supports min/max postfix: “audio.in.jitter.max < 500” or “audio.in.jitter.min > 10”

Jitter Bufferaudio | videoin | outYesYesEvaluate the jitter buffer size (based on jitterBufferMs getstats metric)
“audio.in.jitterbuffer < 200”
Codecaudio | videoin | outYesYesEvaluate the codec name
“audio.out.codec.channel == ‘OPUS'”
“audio.out.codec == ‘opus,vp8′”
fpsvideoin | outYesYesEvaluate frames per second

 

  • “video.in.channel.fps > 25” (calculate value from googFrameRateDecoded getstats metric)
  • “video.out.channel.fps > 25” (calculate value from googFrameRateSent getstats metric)
  • Also supports min/max postfix: “video.out.channel.fps.max < 32” or “video.out.channel.fps.min > 25”
Widthvideoin | outYesYesEvaluate video resolution width
“video.out.channel.resolution.width > 320”
Heightvideoin | outYesYesEvaluate video resolution height
“video.out.channel.resolution.height > 240”
Time –Evaluate the value of a timer, managed using the function .rtcGetTimer()
Metric –Evaluate the value of a metric, managed using the function .rtcSetMetric()

Code examples:

Listed below are general examples of how you can use test expectations in your code:

// The session was longer than 60 seconds
.rtcSetTestExpectation("connectionDuration > 60")

// We have both an incoming and an outgoing channel
.rtcSetTestExpectation("audio.in >= 1")
.rtcSetTestExpectation("audio.out >= 1")
.rtcSetTestExpectation("video.in >= 1")
.rtcSetTestExpectation("video.out >= 1")

//we have no empty channels
.rtcSetTestExpectation("audio.in.empty == 0")
.rtcSetTestExpectation("audio.out.empty == 0")
.rtcSetTestExpectation("video.in.empty == 0")
.rtcSetTestExpectation("video.out.empty == 0")

.rtcSetTestExpectation("audio.in.bitrate >= 35")
.rtcSetTestExpectation("audio.in.bitrate <= 45")
.rtcSetTestExpectation("audio.in.bitrate.max <= 50", "warn")

.rtcSetTestExpectation("audio.out.bitrate >= 35")
.rtcSetTestExpectation("audio.out.bitrate <= 45")
.rtcSetTestExpectation("audio.out.bitrate.max <= 50", "warn")

.rtcSetTestExpectation("audio.in.channel.bitrate > 35")
.rtcSetTestExpectation("audio.out.channel.bitrate > 35")
.rtcSetTestExpectation("audio.in.channel.bitrate < 45")
.rtcSetTestExpectation("audio.out.channel.bitrate < 45")

//Packet Loss is less than 2%
.rtcSetTestExpectation("audio.in.packetloss  <= 2")
.rtcSetTestExpectation("audio.out.packetloss  <= 2")

.rtcSetTestExpectation("audio.in.channel.packetloss  <= 2")
.rtcSetTestExpectation("audio.out.channel.packetloss  <= 2")

.rtcSetTestExpectation("audio.in.channel.data >= 6000")
.rtcSetTestExpectation("audio.out.channel.data >= 6000")

.rtcSetTestExpectation("audio.in.channel.packets>= 600")
.rtcSetTestExpectation("audio.out.channel.packets>= 600")

.rtcSetTestExpectation("audio.out.roundtrip <= 100")
.rtcSetTestExpectation("audio.out.channel.roundtrip <= 100")
.rtcSetTestExpectation("audio.out.roundtrip.max < 150", "warn")

.rtcSetTestExpectation("audio.in.jitter < 200")
.rtcSetTestExpectation("audio.in.channel.jitter < 90")
.rtcSetTestExpectation("audio.in.jitter.max < 120", "warn")

.rtcSetTestExpectation("audio.in.codec == 'OPuS'")
.rtcSetTestExpectation("audio.out.codec == 'OPUS'")

//adding video channel checks
.rtcSetTestExpectation("video.in.bitrate >= 200")
.rtcSetTestExpectation("video.in.bitrate < 500")
.rtcSetTestExpectation("video.in.bitrate.max <= 650", "warn")
.rtcSetTestExpectation("video.in.bitrate.min >= 150", "warn")

.rtcSetTestExpectation("video.out.bitrate >= 200")
.rtcSetTestExpectation("video.out.bitrate < 500")
.rtcSetTestExpectation("video.out.bitrate.max <= 650", "warn")
.rtcSetTestExpectation("video.out.bitrate.min >= 100", "warn")

.rtcSetTestExpectation("video.in.channel.bitrate >= 200")
.rtcSetTestExpectation("video.out.channel.bitrate >= 200")
.rtcSetTestExpectation("video.in.channel.bitrate < 500")
.rtcSetTestExpectation("video.out.channel.bitrate <500")

//Packet Loss is less than 2%
.rtcSetTestExpectation("video.in.packetloss  <= 2")
.rtcSetTestExpectation("video.out.packetloss  <= 2")

.rtcSetTestExpectation("video.in.channel.packetloss  <= 2")
.rtcSetTestExpectation("video.out.channel.packetloss  <= 2")

.rtcSetTestExpectation("video.in.channel.data >= 7000")
.rtcSetTestExpectation("video.out.channel.data >= 7000")

.rtcSetTestExpectation("video.in.channel.packets >= 700")
.rtcSetTestExpectation("video.out.channel.packets >= 700")

.rtcSetTestExpectation("video.out.roundtrip <= 100")
.rtcSetTestExpectation("video.out.channel.roundtrip < 100")
.rtcSetTestExpectation("video.out.roundtrip.max < 120", "warn")

.rtcSetTestExpectation("video.in.jitter < 100")
.rtcSetTestExpectation("video.in.channel.jitter < 100")
.rtcSetTestExpectation("video.in.jitter.max < 120", "warn")

.rtcSetTestExpectation("video.in.codec == 'vp8'")
.rtcSetTestExpectation("video.out.codec == 'vp8'")

.rtcSetTestExpectation("video.in.fps <= 35")
.rtcSetTestExpectation("video.out.fps <= 35")
.rtcSetTestExpectation("video.in.fps >= 25")
.rtcSetTestExpectation("video.out.fps >= 25")

.rtcSetTestExpectation("video.in.channel.fps>= 10")
.rtcSetTestExpectation("video.out.channel.fps>= 10")

.rtcSetTestExpectation("callSetupTime < 50 ")

.rtcSetTestExpectation("audio.in.channel.bitrate.drop < 50", "warn")
.rtcSetTestExpectation("audio.out.channel.bitrate.drop < 50", "warn")
.rtcSetTestExpectation("video.in.channel.bitrate.drop < 50", "warn")
.rtcSetTestExpectation("video.out.channel.bitrate.drop < 50", "warn")

Below is an example of how an error will show in the report:

.rtcSetTestExpectation("audio.in.packets < 2500")
p8RJTbXd-8Jh-1HHDojmHuPJHobt6AAj9w.png

The following example shows how expectations can be used based on scope. In this case, different expectations for different probes in the same test run:

if (agentType === 1) {
  client.rtcSetTestExpectation("video.out.channel.bitrate > 0")
  // video.OUT.channel.bitrate > 0
  // will be evaluated only for the code here
}
else // agentType is not 1
{
  client.rtcSetTestExpectation("video.in.channel.bitrate > 0")
  // video.IN.channel.bitrate > 0
  // will be evaluated only for the code here
}

Was this article helpful?

Related Articles

Leave a Reply 2 comments

David Pautler - February 1, 2019 Reply

My test result was “Agent 1: Failed test expectation [audio.in == 1], Actual [0]”. I came to this page to find more explanation of this problem and what I can do about it.

FWIW, my guess is that signalling succeeded but that negotiation for media (encrypted) failed.

The test did succeed yesterday when there was no throttling.

    Tsahi Levent-Levi - February 1, 2019 Reply

    This does indicate that you were expecting a single incoming audio channel and got none.

    The issue might not be in the packet loss in this case. If you look at the webrtc-internals dump (download it), there’s nothing there. Just getusermedia and no peerconnections. It breaks before even trying.

Leave a Reply: