Knowledge Base

  1. Home
  2. Knowledge Base
  3. Technical Support
  4. Script Commands
  5. rtcSetTestExpectation()
rtcSetTestExpectation()

Indicate expected outcome for the specific probe execution. This is used to decide if a test has successfully passed or failed.

The condition is evaluated at the end of the test, based on collected information and metrics. The command can be placed anywhere in the script and can appear multiple times with different constraint values.

Arguments:

Name Type Description
criteria string The criteria to test. See below for the available options
message string Message to invoke if criteria isn’t met
level string Level of expectation:

  • “error” – error occured – fail the test
  • “warning” – consider this as a warning
  • Default value: error

Criteria

A criteria is comprised of the metric to test, an operator and a value.

For example: “video.in > 0” will evaluate that the number of incoming video channels is greater than 0.

Operators

The available operators for the criteria are:

  • ==
  • >
  • <
  • >=
  • <=
  • !=

Criteria metrics

The criteria is defined as a chained definition of the object we wish to evaluate.

The chain hierarchy is:

Media type (audio / video) . Direction (in / out) . channels . Evaluated criteria

Where the channels level is optional, i.e. both formats will work:

  1. video.in.bitrate – Evaluates the total bitrate of all incoming video channels
  2. audio.out.channel.bitrate – Evaluates the bitrate of each outgoing audio channel separately
Evaluated criteria Media type Direction Global (all channels) Per channel Description and example
Connection duration Evaluate the duration of the whole peer connection session in seconds
“connectionDuration > 60”
Number of channels audio | video in | out Yes No Evaluate the number of channels
“video.out >= 1”
Channel is empty audio | video in | out Yes Yes Evaluate if the total number of sent/received packets is zero
“video.in.empty == 0”
Bitrate audio | video in | out  Yes  Yes “audio.out.bitrate >= 3” – evaluate the TOTAL aggregated bitrate of all outgoing audio channels
“audio.out.channel.bitrate > 3” – evaluate the value of EACH outgoing audio channel’ bitrate (evaluating all channels, per channel)

Also supports min/max postfix: “video.out.bitrate.max < 500” or “audio.in.bitrate.min > 30”

Packet loss audio | video in | out Yes Yes Evaluate channels’ packet loss (in %)
“packetloss < 2”
Packets lost audio | video in | out Yes Yes Evaluate the number of packets lost on the channel
“packetslost < 20”
Data audio | video in | out Yes Yes Evaluate total amount of sent/received bytes in the whole session
Packets audio | video in | out Yes Yes Evaluate the total number of packets sent/received
Roundtrip audio | video in | out Yes Yes Evaluate the average round trip time

Also supports min/max postfix: “video.out.roundtrip.max < 200” or “audio.in.roundtrip.min > 200”

Jitter audio | video in | out Yes Yes Evaluate the average jitter value on the network (based on jitterReceived getstats metric)
“audio.in.jitter < 200”

Also supports min/max postfix: “audio.in.jitter.max < 500” or “audio.in.jitter.min > 100”

Jitter Buffer audio | video in | out Yes Yes Evaluate the jitter buffer size (based on jitterBufferMs getstats metric)
“audio.in.jitterbuffer < 200”
Codec audio | video in | out Yes Yes Evaluate the codec name
“audio.out.codec.channel == ‘OPUS'”
“audio.out.codec == ‘opus,vp8′”
fps video in | out Yes Yes Evaluate frames per second

  • “video.in.channel.fps > 25” (calculate value from googFrameRateDecoded getstats metric)
  • “video.out.channel.fps > 25” (calculate value from googFrameRateSent getstats metric)
  • Also supports min/max postfix: “video.out.channel.fps.max < 32” or “video.out.channel.fps.min > 25”
Resolution video in | out Yes Yes Evaluate video resolution
“video.in.channel.resolution == ‘640×480’ “
Time  – Evaluate the value of a timer, managed using the function .rtcGetTimer()
Metric  – Evaluate the value of a metric, managed using the function .rtcSetMetric()

Code examples:

Listed below are general examples of how you can use test expectations in your code:

// The session was longer than 60 seconds
.rtcSetTestExpectation("connectionDuration > 60")

// We have both an incoming and an outgoing channel
.rtcSetTestExpectation("audio.in >= 1")
.rtcSetTestExpectation("audio.out >= 1")
.rtcSetTestExpectation("video.in >= 1")
.rtcSetTestExpectation("video.out >= 1")

//we have no empty channels
.rtcSetTestExpectation("audio.in.empty == 0")
.rtcSetTestExpectation("audio.out.empty == 0")
.rtcSetTestExpectation("video.in.empty == 0")
.rtcSetTestExpectation("video.out.empty == 0")

.rtcSetTestExpectation("audio.in.bitrate >= 35")
.rtcSetTestExpectation("audio.in.bitrate <= 45")
.rtcSetTestExpectation("audio.out.bitrate >= 35")
.rtcSetTestExpectation("audio.out.bitrate <= 45")

.rtcSetTestExpectation("audio.in.channel.bitrate > 35")
.rtcSetTestExpectation("audio.out.channel.bitrate > 35")
.rtcSetTestExpectation("audio.in.channel.bitrate < 45")
.rtcSetTestExpectation("audio.out.channel.bitrate < 45")

//Packet Loss is less than 2%
.rtcSetTestExpectation("audio.in.packetloss  <= 2")
.rtcSetTestExpectation("audio.out.packetloss  <= 2")

.rtcSetTestExpectation("audio.in.channel.packetloss  <= 2")
.rtcSetTestExpectation("audio.out.channel.packetloss  <= 2")

.rtcSetTestExpectation("audio.in.channel.data >= 6000")
.rtcSetTestExpectation("audio.out.channel.data >= 6000")

.rtcSetTestExpectation("audio.in.channel.packets>= 600")
.rtcSetTestExpectation("audio.out.channel.packets>= 600")

.rtcSetTestExpectation("audio.out.channel.roundtrip <= 200")

//adding video channel checks
.rtcSetTestExpectation("video.in.bitrate >= 200")
.rtcSetTestExpectation("video.in.bitrate < 500")
.rtcSetTestExpectation("video.out.bitrate >= 200")
.rtcSetTestExpectation("video.out.bitrate < 500")

.rtcSetTestExpectation("video.in.channel.bitrate >= 200")
.rtcSetTestExpectation("video.out.channel.bitrate >= 200")
.rtcSetTestExpectation("video.in.channel.bitrate < 500")
.rtcSetTestExpectation("video.out.channel.bitrate <500")

//Packet Loss is less than 2%
.rtcSetTestExpectation("video.in.packetloss  <= 2")
.rtcSetTestExpectation("video.out.packetloss  <= 2")

.rtcSetTestExpectation("video.in.channel.packetloss  <= 2")
.rtcSetTestExpectation("video.out.channel.packetloss  <= 2")

.rtcSetTestExpectation("video.in.channel.data >= 7000")
.rtcSetTestExpectation("video.out.channel.data >= 7000")

.rtcSetTestExpectation("video.in.channel.packets >= 700")
.rtcSetTestExpectation("video.out.channel.packets >= 700")

.rtcSetTestExpectation("video.out.channel.roundtrip <= 2000")

.rtcSetTestExpectation("audio.in.codec == 'OPuS'")
.rtcSetTestExpectation("audio.out.codec == 'OPUS'")

.rtcSetTestExpectation("video.in.codec == 'vp8'")
.rtcSetTestExpectation("video.out.codec == 'vp8'")

.rtcSetTestExpectation("audio.in.channel.jitter < 300")
.rtcSetTestExpectation("audio.out.channel.jitter < 300")

.rtcSetTestExpectation("audio.in.jitter < 200")
.rtcSetTestExpectation("audio.out.jitter < 200")

.rtcSetTestExpectation("video.in.channel.jitter < 300")
.rtcSetTestExpectation("video.out.channel.jitter < 300")

.rtcSetTestExpectation("video.in.jitter < 300")
.rtcSetTestExpectation("video.out.jitter < 300")

.rtcSetTestExpectation("video.in.fps <= 35")
.rtcSetTestExpectation("video.out.fps <= 35")
.rtcSetTestExpectation("video.in.fps >= 25")
.rtcSetTestExpectation("video.out.fps >= 25")

.rtcSetTestExpectation("video.in.channel.fps>= 0")
.rtcSetTestExpectation("video.out.channel.fps>= 0")

.rtcSetTestExpectation("video.out.channel.resolution >= '320x240' ", "warn")

.rtcSetTestExpectation("callSetupTime < 50 ")

.rtcSetTestExpectation("audio.in.channel.bitrate.drop < 50", "warn")
.rtcSetTestExpectation("audio.out.channel.bitrate.drop < 50", "warn")
.rtcSetTestExpectation("video.in.channel.bitrate.drop < 50", "warn")
.rtcSetTestExpectation("video.out.channel.bitrate.drop < 50", "warn")

 

Below is an example of how an error will show in the report:

.rtcSetTestExpectation("audio.in.packets < 2500")

p8RJTbXd-8Jh-1HHDojmHuPJHobt6AAj9w.png

 

The following example shows how expectations can be used based on scope. In this case, different expectations for different probes in the same test run:

if (agentType === 1) {
  client.rtcSetTestExpectation("video.out.channel.bitrate > 0")
  // video.OUT.channel.bitrate > 0
  // will be evaluated only for the code here
}
else // agentType is not 1
{
  client.rtcSetTestExpectation("video.in.channel.bitrate > 0")
  // video.IN.channel.bitrate > 0
  // will be evaluated only for the code here
}

 

Was this article helpful?

Related Articles

Leave a Reply 2 comments

David Pautler - February 1, 2019 Reply

My test result was “Agent 1: Failed test expectation [audio.in == 1], Actual [0]”. I came to this page to find more explanation of this problem and what I can do about it.

FWIW, my guess is that signalling succeeded but that negotiation for media (encrypted) failed.

The test did succeed yesterday when there was no throttling.

    Tsahi Levent-Levi - February 1, 2019 Reply

    This does indicate that you were expecting a single incoming audio channel and got none.

    The issue might not be in the packet loss in this case. If you look at the webrtc-internals dump (download it), there’s nothing there. Just getusermedia and no peerconnections. It breaks before even trying.

Leave a Reply: