All Posts by Tsahi Levent-Levi

0 testRTC August 2021 Release Notes

testingRTC & upRTC

Analysis

• High level charts of test results have been “modernized” to support our new charting scheme, where we show bitrates versus quality related metrics – same as we do on the probe level

• We’ve revamped the channel view to make it easier to read. When you double click a channel – this is what you’re going to see now:

• Events log now parses SDP to make these messages easier to read

Nightwatch

We’ve upgraded the Nightwatch version used for scripting to the latest – 1.7.7.

This in turn, enables us to offer better scripting for your tests.

As a first step, you can now use the Nightwatch Testing Library for simpler DOM testing and manipulation.

Here’s how this affect the Jitsi Sample:

Left-hand side shows the code using the Nightwatch Testing Library.

Testing

• H.264 video codec is now supported when testing on Firefox browsers
• Additional environment variables were added to make scripting easier
• We’ve introduced process.env.RTC_EXTRA_TABS to make it simpler to handle switching windows and tabs in testRTC (especially when screen sharing)
• We now show the public IP address of our probe as part of the results

qualityRTC

• Analytics dashboard now shows traffic light colors for the collected tests, making them easier to review:

• CALL QUALITY and VIDEO QUALITY tests now issue a warning if connection occurred over TCP or TLS
• Video tests can now be conducted using virtual cameras instead of real ones
• We now optionally enable canceling tests that are being conducted. This is useful when you need to run qualityRTC on a touch based console with no keyboard connected. Learn more about qualityRTC touch support

• qualityRTC invites are now more manageable, making tracing creator and expiration dates of invites easier from the dashboard

probeRTC

• Added the ability to compare results of two separate probes
• A new RESTART button is now available for running probes. This is useful mainly when using the installed probeRTC daemon applications

watchRTC

We’ve made tremendous progress with our watchRTC product. Here are some of the main improvements:

watchRTC SDK

The JS SDK has gone through some polishing, making it a lot more flexible and powerful to use

• We made it easy to integrate with both your own infrastructure and third party SDKs
• During this process, we’ve started adding proper release notes to the SDK’s installation page

• If you are running in a PWA and conduct sessions on different rooms one after the other, you can now easily modify the roomId and peerId dynamically
• You can now enable and disable data collection dynamically via the SDK
• You can now collect user’s rating and pass it on via the SDK by calling watchRTC.setUserRating()
• Distinguishing weeks and understanding day of week in the Highlights graphs 

Highlight Charts

• We now show striped graphs to make it easy to see weekly trends
• When you hover your mouse over specific bars, it now also indicates the day of the week for the value you are reviewing

Trends Charts

There are now new trends charts available in watchRTC. These offer a closer look at the KPIs and quality metrics on your sessions.

Things we report here include:

• Active sessions duration
• Setup time
• Negotiation errors
• ICE connection failure rate
• Operating systems
• Browsers
• Bitrate (in and out)
• Packet loss and RTT

Drilldown

• You can export room level data to CSV format
• We now show RTT information in the top ribbon of the rooms and peers:

• Where possible, we now indicate the network type used in sessions conducted:

Other

• watchRTC now supports proxy data collection to ease on the need to handle firewall configurations
• You can set user ratings for a certain session via new REST API: PUT /watchrtc/room/{roomId}/peer/{peerId}/user-rating
• Added machine performance information to REST API: GET ​/testagents​/{testAgentId}

0 Using Call end information to debug WebRTC connectivity issues

High level charts of test runs include a yellow Call end vertical line in them. Here’s what it does and what you can use it for.

What is Call end?

The Call end vertical line indicates the time in the test when one of the probes had its WebRTC peer connection closed.

In a stress test, your usual expectation would be to see the Call end indicator close(r) to the end of the test – since you’ll probably be ramping up the number of probes joining the session and then near the end of the test you might want to shut them down gracefully.

If the Call end line appears sooner than expected, then there might be an issue.

How can you use it?

Here is a short analysis of a session we recently looked at:

Key points to remember here:

• Call end happens too soon in the test
• We can see incoming bitrate and jitter growing “out of line” just before that point in time. Probably due to a resources issue 👉 it might make sense to also check memory and CPU use on the probe side to see if you need to optimize your JS code
• To figure out which probe ended the call, you can export the results to CSV
• Focus there on Connection duration column and sort it
• You can also try to plot it on a line chart
• This will give you the probes you need to focus on to figure out what’s wrong

0 What identifiable information watchRTC SDK collects?

watchRTC is focused on collecting metrics, events and KPIs that are important to analyzing and troubleshooting network and behavior issues in your WebRTC application.

Since our SDK gets integrated into your application, we make it a point to collect as much as we possibly can to help you with your monitoring needs while at the same time we keep the privacy of your users.

When collecting information, there are only two types of fields that can be considered PII in watchRTC:

1. IP addresses
2. Peer ID’s

While we do collect IP addresses, we don’t fully store them. We store only the 3 first octets of the 4. An IP address of 83.130.194.181 for example will be stored and displayed as 83.130.194.x. Before we “obfuscate” the IP address, we also geolocate the user, to understand his country and city. While doing that, we reduce the granularity of longitude and latitude to ~50km radius.

Peed ID’s are information you determine. It is advisable not to use emails or other identifiable information there and instead use them to correlate with your own application ID’s for the peers in the session.

It should also be noted that the communication between the SDK and our backend servers is encrypted in transit.

0 How to use Nightwatch Testing Library with testRTC?

Also known as NWTL, the Nightwatch Testing Library is a powerful utility library for Nightwatch that makes figuring out and finding the elements in your UI that needs to be clicked, filled or waited upon clearer and cleaner. As this library states, it offers “simple and complete custom queries for Nightwatch that encourage good testing practices”.

We’ve see an good number of our clients interested in this, so we added support for it as part of our scripting capabilities.

In order to use this library in your test scripts, add the following statements at the beginning of your test script:

const { getQueriesFrom } = require("@testing-library/nightwatch");Code language: JavaScript (javascript)

This will load the NWTL as part of your script and enable you to use its capabilities.

Here’s a comparison of how the Jitsi sample script would be written if we used NWTL in it:

0 Using qualityRTC in keyboardless touch consoles

qualityRTC can work quite well in touch-based environments where the use of a keyboard is impossible or not desired.

To get that done properly, you can use the URL parameters available in qualityRTC in order to prefill the online form.

If needed, two additional customizations are available for such consoles:

1. Support for smaller screen resolutions and both landscape and portrait display modes
2. Ability to add a CANCEL button to the test, when the user needs to cancel or rerun a test without the need to refresh the page or wait for the current test to finish

Cancelled tests appear in the history dashboard with the partial data collected during the test.

0 Setting up a proxy for watchRTC traffic

watchRTC uses a websocket connection from the users to our servers in order to collect the WebRTC statistics and telemetry data necessary for monitoring your service.

At times, you may want to use an intermediary proxy which the client devices connect to instead of directly to our servers. The main reason for that is when you want control over the IP addresses customers need to open up and configure in their firewalls and wish to keep these addresses limited to your pool.

For that purpose, watchRTC offers the ability to work via a proxy URL, as well as other firewall configuration alternatives.

Architecture

By default, the watchRTC SDK connects directly to the watchRTC backend service via the watchRTC relay.

The diagram below illustrates what happens when you add your own watchRTC Proxy component:

In such a configuration, the watchRTC SDK will communicate with the watchRTC Proxy. The watchRTC Proxy will then send the data it receives from the SDK to the watchRTC backend service via the watchRTC relay.

This gives you the flexibility of deploying the watchRTC Proxy on your own servers and add another layer of control over the data being routed.

The data being proxied uses secure websockets and secure HTTPS connections.

Install your proxy server

You can install and use any websocket proxy server.

nginx has such a websocket proxy, along with a Docker configuration of one.

Other proxies should work just as well.

Make sure the proxy server can handle the amount of concurrent users you plan on working with, as these will now rely on your own infrastructure.

Configure the proxy server

The proxy server installed should redirect incoming websocket traffic to one of these addresses:

1. wss://watchrtc.testrtc.com if you are running watchRTC from our production environment
2. wss://watchrtc-staging2.testrtc.com if you are running watchRTC from our staging environment

In both cases, be sure to also configure HTTPS to be proxied. When a websocket is unavailable, the watchRTC SDK will send its messages via HTTPS.

Passing IP’s when there are Proxies

When using a proxy server location, the standard for passing IPs is to use the ‘X-Forwarded-For’ header

x-forwarded-for: "180.166.146.241, 43.176.32.192"

• When a client connects directly to a server, the client’s IP address is sent to the server (and is often written to server access logs).
• If a client connection passes through any ‘forward’ proxies, the server only sees the final proxy’s IP address.

To provide a more-useful client IP address to the server, the ‘X-Forwarded-For’ request header is used.

Configure the SDK

To use the proxy, add in your watchRTC.init() call under proxyURL the server address you are using:

watchRTC.init({
  proxyUrl: "wss://my.domain.com:8080/websocket"
});Code language: JavaScript (javascript)

0 Private article

0 URL parameters in probeRTC

probeRTC is meant to run on its own with little assistance from the user. From time to time, there is a need to modify its behavior, especially when further troubleshooting is required. This is enabled either via the probe creation options or via URL parameters when running a probeRTC probe.

Supported URL variables

All variables are optional. You are not mandated to add any of them to the URL.

0 Twilio Video React App sample test script

Twilio Video React App is an open source application provided by Twilio as a reference to those developing video based application. The code is maintained on github (here).

If you are using this as a starting point, then the script below is a great way to introduce testing automation to the Twilio Video React framework you are using.

Preparation

Deploy the Twilio application as explained on the github page of the project.

Out of the box, you should get something like this in terms of user experience:

Using the test script

In testRTC, create a new test script:

1. Copy the code from the bottom of this article to your test script (or use the existing sample in your account)
2. Decide the number of probes you want to use
   1. 2 Concurrent probes will be good starting point
   2. Set the Session size to the number of concurrent probes for this sample
3. Replace the Service URL of the script with the URL where your Twilio Video React App server is installed

Test execution

Run the script. It does everything for you.

If you want, you can join the same room from your browser once you see the URL in the progress messages of the test.

Test script code

var agentName = process.env.RTC_AGENT_NAME;
var agentSession = Number(process.env.RTC_SESSION_IDX);
var agentSessionName = process.env.RTC_SESSION_NAME;
var agentNumber = process.env.RTC_AGENT_NUM;

var sec = 1000;
var roomUrl = process.env.RTC_SERVICE_URL;

client
    .pause(agentNumber * 500);

client
    .rtcInfo("testRTC agent start - agent: %s room: %s", agentName, roomUrl)
    .url(roomUrl)
    .waitForElementVisible('#input-user-name', 120 * sec)
    .pause(300) //wait for page render 
    .setValue('#input-user-name', agentName.replace(/-/g, ''))
    .pause(300)
    .setValue('#input-room-name', agentSessionName.replace(/-/g, ''))
    .click(".MuiButtonBase-root.MuiButton-root")
    .pause(5000)

    .useXpath()
    .waitForElementVisible("//*[text()='Join Now']", 120 * sec)
    .pause(300)
    .click("//*[text()='Join Now']")
    .rtcProgress("In  " + roomUrl)

    // Give some time to collect media stats
    .pause(60 * sec)
    .rtcScreenshot('mid-call')
    .pause(60 * sec);Code language: JavaScript (javascript)

0 testRTC June 2021 Release Notes

Testing & Monitoring

Analysis

• We’ve updated the ribbot at the top of test results
  • Added performance (CPU and memory use of probes)
  • Added RTT (round trip time)
  • Removed call setup time, now part of the overview section

• Our Advanced WebRTC Analytics on test results now shows metrics also for the selected candidate pair. This allows us to view the bandwidth estimation of WebRTC (availableOutgoingBitrate)

• We’ve added channels start time and duration information, making it easier to analyze results, especially in large tests

• Machine performance graphs are now synchronized to the same timeline and starting point of all other graphs making it easier to compare results
• Uploading a webrtc-internals dump file has a simpler UI now that allows for both drag & drop as well as uploading files

Testing

• We all have our own favorites. Some of us have favorites in the test scripts they wish to run using testRTC. So we’ve enabled you all to pick your favorites 👍👍👍

• Added a new script command .rtcSetCustomExpectation() and .rtcSetCustomEventsExpectation() that can be used if you’d like to add an expectation that conducts a specific calculation that we don’t offer out of the box using .rtcSetTestExpectation()

• Added a new script command .rtcWaitForExternalValue() which works with a newly introduced REST API /testruns/externalValue. Using this enables sending external values into a running test scripts. What can this be used for? Dealing with 2FA requirements that need to glean information from an SMS or an email that is intercepted outside of the test script itself. There are probably other uses for it as well 😉
• Added new run options #probe-screen-width and #probe-screen-height to enable changing the screen resolution. This is useful in screen sharing testing
• Webhooks are now invoked after the test run is fully completed and has been analyzed instead of when the test completed. This was done to prevent a race condition when trying to access metrics results “too soon”

Monitoring

• Added REST API /monitors/{monitorId} to retrieve list of last monitor run id’s
• Added REST API /monitors/{monitorId}/last to retrieve the status of the last monitor run

qualityRTC

qualityRTC invites

qualityRTC now includes a new invite system in the Enterprise plan. This enables the creation of specialized invites for conducting network tests instead of having a single generic link.

The invite system enables you to create special links that are:

• Time limited
• Count limited (the number of times they will “run”)
• Prefilled fields
• Include new custom fields

This can be used to:

• Restrict the use of the network tests without an explicit invite
• Share an invite with a large user base of a specific account/client of yours
• Collect information from specific users (agents of a contact center, asking them about their location for example – WFH or office)
• Provide access to invites to partners

Invites can be created via a new API call as well as directly from the dashboard:

When users use the invite link, the results appear both in the History section in qualityRTC and in the invite page itself for easy access. The results table can also be embedded as an iframe elsewhere programmatically if needed.

Supported infrastructure

We’ve had quite a few clients onboarding our qualityRTC service recently, which got us working hard on additional integrations. Instead of recounting what we’ve added, here’s the list of what we now support out of the box:

You can find more details in our qualityRTC integrations article.

Widgets improvements

• VIDEO BANDWIDTH, VIDEO P2P and VIDEO QUALITY tests now and share in the log the connection type (this will show if connection is via TURN and what transport is used – UDP, TCP or TLS)

• VOICE TEST tests can now be configured to not use the local microphone and generate audio automatically instead. This can increase privacy as well as enable testing using devices that don’t have a microphone connected to them
• TURN CONNECTIVITY widget now offers a simplified UI option available, where the test widget only shows if it has connectivity and with what protocol (prioritizing UDP over TCP and TCP over TLS). All measurements collected still appear in the log for your support team

• TURN CONNECTIVITY widget now supports the ability to conduct tests on separate TURN server deployments. This can be used when you want to compare different TURN servers (such as your own in-house ones versus a managed service one such as XirSys or Twilio)

• DNS LOOKUP now distinguishes between blocked connections to timeout events in the log
• The silent camera test can now optionally check for camera resolutions as well

Miscellaneous improvements

• Multilingual support is now available for the frontend page the end users view. Current languages include: English, French, Spanish, Portuguese. You can learn more here
• Alert emails for qualityRTC now support multiple email addresses that can be alerted at once
• Log access in qualityRTC results can be restricted to the support person, making it unavailable for the end user who executed the test. This is desirable if you want to keep the results as simple as possible for your users
• Our on premise servers for speed testing and TURN now include management interfaces for easier maintenance (this includes heartbeat, metrics collection, logging and high availability configuration).
• You can now embed the frontend UI as an iframe. For this, you can also use ?/embedded=true as a URL parameter to remove the header and footer of the page

probeRTC

You can now access the history results of probeRTC and not just the last run. At the end of the probe’s results page, there’s now a HISTORY button you can click to access a new detailed table:

Printing results of probeRTC was impossible. This got fixed.

watchRTC

A slew of improvements were added to watchRTC, and we are now ready for launch it to the general public – still as beta, but not private anymore. If you’d like to join – let us know.

Things we’ve done:

• Added a new highlights dashboard, sharing high level information about the recent runs. This can be filtered by dates as well as by tags

• Included unsuccessful connections to peer tables
• Polished results view