All Posts by Tsahi Levent-Levi

0 testRTC February 2021 Release Notes

Account Settings

We are starting to flesh out the user-facing account settings and profile settings. As such, you will see a new sidebar menu:

We are starting to build up the account configuration and user settings area of testRTC, starting with the testing and network testing products.

Here’s what you are now able to do through the settings screen:

1. Update your API key
2. Configure qualityRTC’s email delivery, webhook and password
3. See who has access to the project

Webhooks

As we expand our user base and products portfolio, we get more and more requests for integrations. These include the need to provide webhook support.

We decided in this release to streamline and standardize the way we invoke webhooks and to offer richer formatting options. These now include templating, AWS SNS and CloudEvents among other options.

Check out our webhook formatting options.

Testing & Monitoring

Analysis

We’ve overhauled our graphs on the single probe results. The new view is a lot more powerful. Here’s a quick overview of what that means:

To make things even better, in the Advanced WebRTC Analytics, we’ve gone ahead and polished the interface as well:

• Static metrics that had no real use were removed
• Metrics we thought were less important are now disabled by default
• A toggle button was added to the legend, to make it easy to quickly select one or two metrics and display only them

Stress Testing

• We now ignore #har-file and #pcap run options on tests with more than 4 probes or longer than 10 minutes. These log files are heavy and affect performance, so we make sure to disable them on stress tests
• With large tests there’s a pagination mechanism to go through the probes. The pagination buttons are now placed at the bottom of the table as well as at the top
• We are deprecating #no-internals run option and replacing it with a #webrtc-internals run option:
  • Not setting it will collect webrtc-internals for tests with a #timeout shorter than 30
  • Setting it to true will always collect webrtc-internals
  • Setting it to false will not collect webrtc-internals
• We’ve added the pagination bar for the probes list in test results to both the top and the bottom of the probes list. This makes it easier to navigate through them

Testing

• We now automatically allow using HTML5 Geolocation APIs in tests
• Exporting test results of a single test now provide more metrics
• Exporting test results of multiple tests from the History page now… works
• The API now includes an option to change the session size of a test when running it, making it easier to create programmable variations of tests

Monitoring

Monitoring ribbons have been rewritten to make them more useful.

They now include information about the last 15 days, showing the distribution of daily monitor runs.

qualityRTC

We’ve got a new page on our website for qualityRTC! But I am guessing you’re here for the new capabilities of this product 😉

• When there are any issues found, the log button will show a red indication of that

• In the same spirit, when a specific test has issues found for it, it will list them next to its icon. Clicking the icon on the top left corner of each results widget will get you right to the relevant point in the log
• We’ve made camera and microphone access failures more verbose and explanatory, making them easier to figure out and troubleshoot
• We now collect and log the cores count of the machine running the test
• New silent test: GPS Location. We can now collect the GPS location of the user and place it in the log
• New silent test: Hardware availability. We can now try and figure out if hardware acceleration is used for video calls
• Call Quality test
  • The widget now writes to the log both incoming and outgoing metrics
  • Call Quality widget can now be configured to show one of 3 different layouts:
    1. Inbound only – this is the default view we had until today
       
    2. Combined – shows in a single column the highest value from inbound and outbound metrics for packet loss and jitter
       
    3. Detailed – double column, splitting inbound and outbound
       
  • We now put in the log additional information such as the network type and IP+port pair of the connection
• Location widget
  • Can now be configured to not display the “show on map” link
  • IP insights add-on will also indicate cloud hosted machines
• Privacy
  • Those with BI Analytics enabled can now delete test results
  • We can now automatically scrape email and IP addresses from test results after a configurable number of days
• Entry fields in test can now be hidden if provided via URL parameters
• BI Analytics grid improvements
  • We’ve made searching the table easier with a new global search option at the top of the grid
  • We now shows video metrics as well if these are collected by your configuration
• Clients who have the account field configured can now edit its value in the backend dashboard

probeRTC

Installable service

• We can now offer probeRTC as an installable daemon application
• This works for Windows and Mac

Video graphs

• probeRTC is being adopted by more clients. Up until now, our focus has been call centers, but it seems other types of vendors are in need of this service as well – ones that also handle video calls
• This is why in this release, we’ve added graphs for the video tests that probeRTC can conduct

watchRTC

We are working on a new product for passive monitoring of your users’ WebRTC metrics.

This is still running in private beta with select clients. We are expanding the list of clients we work with on it on a weekly basis – reach out to us if you want to join the beta.

0 How to test screen sharing using testRTC

Like everything else you can do with WebRTC, screen sharing is natively supported by testRTC. There are though a few things you should be aware of and prepare in advance with if you want to test and validate your screen sharing feature with testRTC.

1. Understanding browser tabs in testRTC

When using screen sharing, you are going to move between tabs.

testRTC may or may not open chrome://webrtc-internals tab for its own use, which can confuse certain scripts.

This is why we make use of process.env.RTC_EXTRA_TABS environment variable. You can learn more about switching windows and tabs in testRTC.

2. Use run options to grab the whole screen

You can’t automate the screen picker modal of Chrome. What you can do instead is let Chrome know you wish to override that modal and always capture the whole screen.

To do that, add the following string to your run options:

#chrome-cli:auto-select-desktop-capture-source=Entire screen,enable-usermedia-screen-capturingCode language: PHP (php)

3. Pick a video to share

With WebRTC, a user is likely to click a screen sharing button and then move on to a different tab in his browser or a different application that he is sharing. You’d want to do that as well, to keep the screen that WebRTC encodes as dynamic as possible.

For that purpose, think of a YouTube video that you would want to share. My preference is a movie preview, but that’s not always what you’re looking for. Here are two alternatives we use from time to time:

1. A rather static presentation (this link)
2. Big Buck Bunny – a classic… (this link)

Notice that we use the following URL format for YouTube:

youtube.com/embed/?playlist=<video-id>&autoplay=1&loop=1

• embed causes the video to load in full screen mode
• autoplay will… auto play the video
• loop will cause the video to loop once completed, so you don’t have to worry about its length

4. Tab switching

Once you click the share button on your UI, it is time to open a new and open that YouTube video URL.

Here’s how we do it usually:

function startedSharing() {
    client
        .rtcEvent('Screen Share ', 'global')
        .rtcScreenshot('screen share')

        .execute("window.open('" + videoURL + "', '_blank')")
        .pause(5 * sec)

        // Switch to the YouTube tab
        .windowHandles(function(result) {
            var newWindow;
            newWindow = result.value[1+Number(process.env.RTC_EXTRA_TABS)];
            this.switchWindow(newWindow);
        });
}
Code language: JavaScript (javascript)

A few observations here:

• You can place this function as one of your assets
• Adding an event using rtcEvent() at the beginning comes in handy when you’ll view the resulting graphs in the end
• The videoURL is that YouTube URL we’ve shared earlier. Use a variable for that so it will be easier to search and modify based on your needs
• Actual window switching is done using pure Nightwach commands

If you want/need to switch back and stop screen sharing, you can use this piece of code:

function stoppedScreensharing() {
    client.windowHandles(function (result) {
        var newWindow;
        newWindow = result.value[Number(process.env.RTC_EXTRA_TABS)];
        this.switchWindow(newWindow);
    });
}Code language: JavaScript (javascript)

Just don’t forget to click the stop screen sharing button in your UI afterwards.

5. Select who is going to screen share

Use our sessions mechanism to split your test into sessions (a session is a room usually). In each session, have a single probe screen share and the rest designated viewers.

Here’s how you can do that:

var probeIdx = Number(process.env.RTC_IN_SESSION_ID);
var sec = 1000;

if (probeIdx === 1) {
    // Screen share here
}

client.pause(60*sec);

if (probeIdx === 1) {
    // Stop screen sharing here
}Code language: JavaScript (javascript)

6. Take a screenshot on the viewers’ side

For debugging purposes, you can use rtcScreenshot() to take a screenshot of what gets shared for those who aren’t screen sharing.

0 Webhooks in testRTC

testRTC enables the use of webhooks to catch a rich set of events within the system. These can be used for testing as well as monitoring. These are most often used for things like;

• Notifications
• Run results
• Custom alerts

URL only

If you place in the webhook the URL only, we will be sending it out as a JSON object.

https://dummy.url

JSON object

The webhook configuration below sends the information as a JSON object in the body of the message.

{
   "format": "json/object",
   "url": "https://dummy.url"
}

JSON text format

If you are planning to integrate the webhook with something like Slack or Zapier, then the best approach is to use a JSON text format, where we “stringify” the information.

{
   "format": "json/text",
   "url": "https://dummy.url"
}

You can read more on sending test results via webhook to Slack.

CloudEvents

Here is how to send our webhooks as CloudEvents v1.0 format:

{
   "format": "cloudevent",
   "url": "https://dummy.url"
}

The above will send the webook to “url” formatting the body of the message as CloudEvents format.

Amazon SNS

If you plan on sending the webhook to Amazon Simple Notification Service, then you can use the following format:

{
    "provider": "aws-sns",

    "secretAccessKey": "xxx",
    "region": "xxx",
    "accessKeyId": "xxx",

    "roleArn": "xxx",
    "topicArn": "xxx",

    "body": {
      "Server": "TestRTC",
      "Instance": "<%this.runName%>",
      "Severity": "TestRTC",
      "Message": `testRTC Alert for <%this.testName%> 
      <%this.runName%> 
      error:<%this.error%> 
      url:<%this.resultUrl%> 
      failure:<% this.failureScreenshot %>`,
    }
}

Custom body templating (limited application)

For testingRTC and upRTC, you can also format our webhook as you see fit using our body templating format. Below is an example for it.

{
    "url": "https://testrtc-webhook-director.netlify.app/.netlify/functions/slack-message-director",
    "body": {
      "testRunId": "<%this.testRunId%>",
      "testName": "<%this.testName%>",      
      "userName": "<%this.userName%>",      
      "projectName": "<%this.projectName%>",
      "projectId": "<%this.projectId%>",      
      "status": "<%this.status%>",
      "concurrentUsers": "<%this.concurrentUsers%>",
      "numberOfProbesSuccess": "<%this.numberOfProbesSuccess%>",
      "numberOfProbesWarning": "<%this.numberOfProbesWarning%>",
      "numberOfProbesFailure": "<%this.numberOfProbesFailure%>",
      "totalTestTimeMin": "<%this.totalTestTimeMin%>",
      "score": "<%this.score%>",
      "runName": "<%this.runName%>",
      "runType": "<%this.runType%>",
      "error": "<%this.error%>",
      "additionalInfo": "<%this.additionalInfo%>",     
      "failureReasons": "<%this.failureReasons%>",
      "failureScreenshot": "<%this.failureScreenshot%>",
      "resultUrl": "<%this.resultUrl%>"
    }
  }

Custom variables

You can pass certain variables to the webhook in its advanced format. This is useful for collecting information and when you want to format the message itself.

{
  "url":"https://webhook-destination.com",
  "body":{
    "testRunId":"<%this.testRunId%>",
    "testName":"<%this.testName%>",
    "runName":"<%this.runName%>",
    "status":"<%this.status%>",
    "error":"<%this.error%>"
  }
}

0 What network ports does qualityRTC use?

For its own use, qualityRTC requires port 443 to be open and reachable.

For anything that is WebRTC infrastructure specific, the required ports are similar to those of the service being tested. This is due to the fact that qualityRTC integrates with that infrastructure to conduct its connectivity and network quality checks.

0 testRTC December 2020 Release Notes

New Product: watchRTC in Private Beta

We have now added a new passive monitoring service, which collects metrics and data from real users running your system.

Why use testRTC for this?

• Enjoy the same look and feel, as well as the level of depth and analysis we offer today in our testingRTC and upRTC product – but on your own users’ sessions
• Fast to load, simple to use. Just as you’d expect from our other services

For now, this product is in private beta. If you’d like to join the private beta, contact us.

Facelift

We’ve introduced a new dashboard recently, to better fit the growing number of product lines we now offer at testRTC.

The new dashboard makes the information more accessible and will show data only to the products you’ve licensed, reducing clutter.

We also introduced estimated test runs for qualityRTC and we plan on expanding it to other products soon.

Other than that, we added more breadcrumbs to make navigating back and forth easier.

Testing & Monitoring

Analysis

In recent months we’ve seen an increase in the size of tests our clients want to run. We are working hard these days in improving and beefing up our service:

• Large tests results load slightly faster, using less memory and CPU while doing so
• We are now calculating scores properly when some of the probes fail to connect or collect media results

As in previous releases, we’re making it easier to read and analyze the results:

• Probe results now indicate the session sizes, so you don’t need to browse back
• Channels now show their start time and duration when you expand the channel details

Stress Testing

Large tests are different in nature. We’ve seen many clients using a growing number of probes in their tests recently, and we’ve added a few unique features towards these use cases:

• Best effort. Tests with more than 50 probes in them will run in a new best effort mode. This means that if not all probes were allocated at the beginning of the test (something that has a higher probability of happening the larger the test is), then the test will still execute with a lower number of probes. Learn more here

Testing

• .rtcCallWhenDone() is a new script command that enables you to create a function that will be called when testRTC wraps the test script – even if it fails prematurely
• .rtcCaptureCallData() now allows choosing the tab holding chrome://webrtc-internals. This is useful for restrictive services such as proctoring applications
• .rtcSetMetricFromThresholdTime() can now be used to calculate the time from a given event until metric values reach certain thresholds. This can be used for performance and bandwidth estimation purposes
• .rtcSetTestExpectation() – support for new metrics:
  • We now support machine specific expectations such as CPU, memory and network data
  • We now support more getstats metrics such as NACK and PLI counts
• A new environment variable RTC_SESSION_COUNT can be used to get the number of sessions allocated in a test
• Tests now collect machine performance metrics by default. You can use the run option #perf:false to disable this collection if needed

APIs

We’ve introduced more metrics out of test results so you can now collect and build your own dashboards:

• /testruns/{testId}/?detailed=true now retrieves more metrics data out of a test run
• /testagents/{testAgentId}/files was added. Now you can download the collected test data such as browser log and webrtc-internals dump file
• APIs got polished a bit, some were added a few additional fields

qualityRTC and probeRTC

We’ve got the release notes for these products separately this time, published earlier.

qualityRTC/probeRTC December 2020 release notes

0 rtcSetMetricFromThresholdTime()

Calculate the time it takes for a media related metric value to reach a certain threshold from an event and store that time in a custom metric.

Arguments

Supported criteria

The following criteria are supported:

• [video|audio].[in|out].bitrate – expressed in Kbits
• video.[in|out].fps – expressed in integer values

Code examples

client
  .rtcSetMetricFromThresholdTime("timeTo1Mbps", "CallStart", "video.out.bitrate > 1000", "avg");Code language: JavaScript (javascript)

The example checks the time it took the probe to reach 1 Mbps of outgoing video bitrate, placing it in a custom metric called timeTo1Mbps and aggregating it as average across probes in the test.

0 testRTC December 2020 Release Notes for qualityRTC

The December release here is LARGE, so we’re splitting it into two separate announcements. First one is about qualityRTC and probeRTC – our support facing solutions.

qualityRTC: Network Testing

We are seeing tremendous interest and adoption of qualityRTC, as such, we have been iterating quickly on client requests and feedback for this product.

In general, the documentation for qualityRTC in our knowledge base has been needed up considerably.

New tests

• A new silent DNS Settings test is now available
  • In it, we try to figure out the DNS servers configured for the user
  • The result is written in the log and includes a list of IP addresses, their geographic location and corporation they belong to
• Out-of-the box support for Frozen Mountain LiveSwitch is now available
• We can now collect the device names for microphone and camera and place it in the test log
• We now offer Speed testing machines and TURN servers as on premise installations. If you would like to run accurate tests for your data center, then these are now available

New features

• The Video Quality widget now provides more information, splitting metrics between incoming and outgoing media streams
• Since the LOG is becoming a more important part of the test results, we’ve decided to make its window larger. Expect additional improvements here moving forward
• Test ID now shows on the result page, making it easier to communicate between the end user and your support team
• Email and reason fields can now be customized
  • Clients can change them to anything they see fit (name and case number, etc)
  • Additional fields can be added if needed
• A new configuration enables discouraging search engines from indexing your network testing page
• You can now use ?run= URL parameter to run only specific tests out of the tests conducted. Learn more here
• We have added a new account field that can be used as a URL parameter. Learn more here
• We’ve optimized how we load the qualityRTC page for our users
  • The page is now considerably smaller in size and faster to load
  • We now load only the pieces of tests you need and do that on demand
• We can now configure a webhook for tests conducted in qualityRTC
  • The webhook will be invoked whenever a test was executed, or when certain test result thresholds are met (warnings or faulty results)
  • A JSON blob with all metrics collected is provided (more details here)
  • This means you can now collect and analyze that data within your own BI environment
• Alert thresholds are now available. Up until now, you either received an email on each test conducted or never. You can now receive such an email only if certain threshold conditions are met
• BI/Analytics table has been improved
  • If new tests are available, they will now be indicated on a new refresh button

• Grouping now shows averages on metrics collected. The screenshot below marks in yellow the average values of the specific location that was expanded (there were 3 network tests conducted in that location)

• Export function exports more data, including browser, operating system and device names
• You can now search for a specific network test result based on its id
• Added the IP address, so you can group or search by it

probeRTC: Zero install network probes

• Alert thresholds are now available. When the probe’s results are within the acceptable limits you can now configure it to send out an alert email
• We’ve created a new explainer video about this product