Conducting a WebRTC test is challenging, but the devil is in the details. testRTC’s cloud WebRTC testing and validation platform that scales with ease.
Testing WebRTC applications is challenging. From simple 1:1, through figuring out how to optimize for large group video calls to live broadcasts to thousands of participants or more; what you will immediately notice is the need for simple and powerful automation. Unlike other web technologies, WebRTC brings with it three unique characteristics:
- WebRTC is a resource hog. This translates into complexities in allocating resources when conducting large scale tests
- Highly interactive. A scenario involving multiple people requires multiple people, making scaling tests with a single human tester virtually impossible
- Deciding success or failure using common web metrics is useless. WebRTC comes with its own set of metrics that needs to be gathered and analyzed
With the increasing demand of remote communications and collaboration, the use of WebRTC has significantly grown and skyrocketed. From a niche technology it is now mission critical for many organizations and people around the world.
testRTC has identified this need in 2015, investing in the creation of a self service, best in class platform for testing WebRTC at scale and on demand.
Table of contents
What is a WebRTC test and WebRTC testing?
WebRTC testing is part web part VoIP in nature. On the one hand, you need to test a web-based application. On the other hand, you need to test a VoIP service.
This intersection of worlds brings with it a need to use innovative testing techniques that merge the best of both worlds.
As with any other kind of software testing, you will need to adopt the use of both manual and automated testing.
Manual testing offers the flexibility and intelligence of humans. This makes a lot of sense in WebRTC application augmen testing, simply because humans are the ones that directly interact with the service being tested autm. With manual testing it is important to use tools that enable us to easily collect and analyze WebRTC metrics. These speed up the testing process as well as make it more effective.
Automated testing offers the power of speeding up the testing process considerably and to evaluate scenarios that are impossible to achieve with manual testing, such as instantiating and validating hundreds or thousands of participants in a session. Without the use of automated testing, you end up relying mostly on your users to tell you what is broken with your application.
How to test WebRTC?
You can conduct WebRTC tests manually or in an automated fashion. In both approaches, there are a lot of moving parts you need to take into account.
Device resources
The most obvious aspect of testing WebRTC is the client side. WebRTC runs on various browsers and devices. Depending on your application, you will need to write down the list of combinations of devices your users might be using and focus on testing on these devices.
On each device, make sure to look at the CPU and memory consumption of your WebRTC application in real life scenarios. This will give you an indication of how your users are going to experience your service.
When growing the number of participants in a single group video call for example, it is important to test these, since in many cases, that is where your bottleneck of scalability is going to be.
👉 Validate that you dynamically change your application’s behavior to cater for the many different devices and the available resources they have.
Server-side infrastructure
With WebRTC, there are different types of servers that are used: signaling servers, media servers and STUN/TURN servers.
You need to make sure to test how these servers operate under stress and over long periods of time, conducting soak tests. Make sure to test for CPS – how many “calls” per second your service can handle. In most cases, trying to connect 1,000 users to your service in the exact same second is going to fail.
Find out what is the breaking point of your infrastructure – it might be CPU utilization, but in many cases, it will be pure network traffic – the number of packets certain servers can process (see also our article: Network Jitter).
👉 Check out below how convenient stress testing can be:
Network conditions
Networks aren’t static. They vary and fluctuate over time. You can start a session with a great network with pristine conditions only to see it degrade a minute later.
Some of the tests you will need to conduct are in these areas exactly. Figuring out if your application can cope with the dynamic nature of the network. Here you should check how quickly your media servers return to high bitrate and high frame rate operation after having some rough time of low bitrate or high packet loss.
Check how does the media server cope with different types of networks – multiple users joining over different network conditions – does each get the treatment and video streams he needs or does the service degrade to lowest common denominator?
👉 Your WebRTC application may work great at peak network performance, but you need to make sure it operates well where packet losses and bandwidth restrictions are a fact of life.
User behavior
There are two aspects here.
The first one, is that no matter the amount of automation and sophistication you plan on using, human behavior with an interactive WebRTC service is going to surprise you. You will need real people to test your service. Sad but true.
That said, there are areas where human testers don’t cut it for your testing. Consider this: In a session of 10 users, each user enters the session at his own laisure. Usually not at exactly the same time. What happens when meetings are larger? Or when your service grows?
There will be times when multiple users will actually try to join simultaneously. At the same time. To the same session. Or to separate sessions.
Test and validate for that. WebRTC services that seem to work really well often break if 100 or more browsers are trying to connect to them at the exact same instance in time.
Tools for WebRTC testing
For the most part, you will need to use more than a single WebRTC testing tool to provide better coverage of the necessary tests.
When you decide on which WebRTC testing tool to use, take into consideration the following aspects of the tool:
#1 – WebRTC traffic generation
First and foremost, a WebRTC testing tool should be able to generate WebRTC traffic.
Since WebRTC relies on external signaling protocols, this can be tricky at times.
Make sure that your testing tool is:
- Capable of supporting the signaling protocol you use for WebRTC
- Generates traffic that is as close as possible to the way your application behaves. This usually boils down to a tool that automates a browser instead of generates its own synthetic, pre-encoded traffic
- Simulates various network conditions, to be able to test your WebRTC media servers and TURN servers properly, and as close as possible to real-life conditions
- Grows dynamically to meet your needs as the size of the tests grow on your end. Especially when you conduct sizing or stress testing
- Up to date when it comes to its WebRTC implementation. WebRTC behavior gets updated once a month or two, with each browser version release. Your WebRTC traffic generation tool needs to keep pace with these changes
Generating traffic is where many of the WebRTC testing tools focus today. This is done in one of two main techniques:
- Using browser automation, building on top of Selenium for example, and relying on the browsers to handle and manage all WebRTC traffic on its own
- Creating synthetic, pre-encoded traffic (or just echoing back what is received). A technique that is used mostly in large scale tests where hundreds of thousands of streams need to be tested and it becomes too expensive to use browsers
#2 – Users simulation
When you generate WebRTC traffic, be sure to think about user behavior as well.
In many cases, WebRTC sessions aren’t symmetric: one user takes the role of a lecturer, while the others are students. In other cases, you may have a few speakers in a discussion that is being streamed to a larger audience.
The tool you use should be powerful enough to handle different roles in a test scenario. Even within a symmetric WebRTC group video call, you will want to test screen sharing, which means there is now a role of a presenter within that group.
Giving roles should be an available option not only in what the user “does”, but also in how the user’s “machine” behaves and of how data is analyzed.
👉 At testRTC we offer powerful role assignment and synchronization mechanisms across the users that join a test, making orchestrating scenarios simple to manage.
#3 – Data collection
Now that we’ve generated traffic in WebRTC, designated roles to the various streams, the time has come to collect the data.
With WebRTC, there are a lot of data points that can be useful:
- Console logs
- WebRTC internal dumps
- WebRTC metrics
- Machine performance statistics
- Screenshots
- …
The more the tool you use is capable of collecting, the higher the possibility that you will be able to pinpoint and troubleshoot potential problems faster.
👉 At testRTC we collect everything and anything that we believe is useful for your debugging and analysis of WebRTC sessions. This include all WebRTC API calls and metrics, as well as console logs, machine performance data, screenshots and even optional trace routes and network log files
#4 – Analysis and visualization of WebRTC metrics
Have you ever tried looking at a webrtc-internals dump file?
This file is the first line of defence developers have in their bug squashing efforts with WebRTC. And yet, reading it is challenging.
Make sure the tool you use makes a better job of analyzing and visualizing WebRTC metrics. Things to look for in a WebRTC testing tool:
- Collect and visualize webrtc-internals dumps
- Ability to collect WebRTC metrics and analyze them without webtrc-internals (they aren’t always available)
- Aggregate as well as drill down analysis of data from large number of users
- Mechanisms to pinpoint outliers and failed streams easily
- Define failure criteria to figure out easily pass/fail status of your tests
- Drill down capabilities, going from a high level view down to the minute details of the results captured
👉 The more data at your disposal, the better suited you will be to debug and solve issues, but the harder it will be to read and analyze the results without a testing tool’s guidance.
#5 – Level of maintenance
As stated earlier, WebRTC is a work in progress. It is always changing.
Be sure that the WebRTC testing tool you use is well maintained and gets frequently updated and upgraded.
Today’s requirements are vastly different than last year’s requirements, as we are working with a technology that is currently in high demand with a lot of market focus and competition.
Things to consider when it comes to the maintenance work necessary for a WebRTC testing environment:
- Upgrading browsers frequently
- Dealing with browser automation tools upgrades (they should be aligned with the browser releases)
- WebRTC depracations, changes and additions
👉 With testRTC these are things you won’t need to be dealing with, as it is part of our job to stay up to date with the most recent developments of WebRTC technology.
Types of WebRTC tests
When conducting tests of WebRTC applications, there are certain areas we need to be aware of – certain test types we should consider covering in our test plan. Here are the major areas.
End-to-end WebRTC testing
In most cases, you won’t need to test the WebRTC implementation itself. That’s up to the browser vendors to handle, along with their own WebRTC interoperability testing.
If you are using third party media servers or a CPaaS vendor, then you may or may not find focusing on testing that layer important to you. I find that different vendors and individuals have different opinions here.
What you should do is make sure to do end-to-end testing. Such tests encompass your whole application logic, and your focus there should be in validating the intersection points between WebRTC and the application logic.
P2P testing
If your application supports peer-to-peer, then there are no media servers to contend with. It makes some of the tests simpler to manage and conduct.
Things to validate with P2P:
- Sizing and stress for your TURN servers and signaling servers
- Different firewall configurations
- For mesh based group scenarios (if you do them), you’ll want to invest a lot of time on testing, polishing and optimizing the experience
Call center WebRTC testing
Call centers require special care and attention when it comes to WebRTC testing. There are many reasons for this:
- One leg of the calls reside in WebRTC while the other in PSTN or SIP
- Stress testing is prime focus, as call centers already operate at scale and just need to validate that WebRTC doesn’t mess things up. This is doubly true for those who transcode Opus to G.711 or G.729 in the process
Stress testing for WebRTC
An important aspect of testing in WebRTC is stress testing. This is done to figure out the answer to one of the following 3 questions:
- How much load can my server handle? This question is asked mainly about media servers and TURN servers
- Does my load balancer work properly? This is the second order question, as many services require to load balance across multiple media servers
- Can my application handle X users/rooms/sessions in parallel? This goes to check the service as a whole
Depending on how you operate your service and its scale, you will be asking one or more of these questions.
If you are interested to learn more on how our clients use our service, read these case studies:
- Testing large scale WebRTC events on LiveSwitch Cloud
- Preparing for WebRTC scale: What Honorlock did to validate their infrastructure
WebRTC group calls testing
A big chunk of what vendors do with WebRTC is group calls. Be it video or voice only calls, these are quite challenging to test. Getting enough real people to join a group call to be able to test it takes time and synchronization, and then being able to figure out and troubleshoot issues is close to impossible – especially when you can’t really know what is the quality of the network or the device the users have.
When dealing with WebRTC group calls, you will need to work hard on optimizing for the size of the group you wish to support – be it 10 users, 49 users in a gallery view, 100 users or larger groups.
Towards that goal, you need to use automation tools capable of the following:
- Stable and predictable machines and networks to run with
- Configurable networks, to test in varying conditions, dynamically controllable by you
- Detailed visual analysis of the results, to ease your understanding of the test results
- Test users coming in from different locations
Live broadcast testing
With live broadcasts you have a small number of broadcasters (publishers) and a larger number of viewers (subscribers). The challenge with many solutions is that they don’t offer video transcoding in the cloud and even if they do, you need to validate that it is being used properly.
In such a case, the following types of tests are necessary:
- Stress the media servers at scale, with many viewers
- Bring viewers from different geographies
- Configure viewers with different network conditions
- Dynamically change network conditions for some of the viewers
What you are looking for here is to understand the overall quality of the solution and to validate that viewers with poor connections don’t cause other viewers to have a poor experience as well.
Mobile testing for WebRTC
Mobile testing in WebRTC poses quite a challenge. The main problem here is the lack of proper automation – using cloud based solutions means that what’s in front of the mobile device’s camera is a static image at best or too dark to be useful. That’s because mobile devices are placed in racks inside data centers for this purpose.
You will be able to automate some of your testing, but the nuanced parts of media quality will be hard to achieve with automation.
There are 3 areas of testing you will need to handle:
- Network. Understanding how mobile networks and devices affect your infrastructure. These types of tests can be simulated and automated without the use of actual mobile devices
- Devices. Test against different devices and brands. Either through cloud automation or crowdsourced testing
- Application. This can use cloud automation in the same way that other mobile applications use
Browser testing
Browser testing with WebRTC can either be done manually or automated locally.
There are 4 main browsers: Chrome, Safari, Firefox and Edge.
Many vendors resort to manual testing of the applications across browsers due to the frequent UI changes they are making or because of the complexity of automating testing of WebRTC across browsers. At testRTC, we offer automated testing for Chrome and Firefox at the moment – especially since our focus to date has been infrastructure testing.
Things to keep in mind while testing browsers:
- Test multiple versions of each browser, as you will bump into them in the wild
- Make sure to test in front of the beta releases to catch upcoming issues
Component testing
WebRTC has many moving parts: browsers, devices, application servers, web servers, TURN servers, media servers. And we haven’t even gotten to listing other infrastructure components such as databases and load balancers.
At times it is important to test each component separately. With WebRTC, that is mostly focused on TURN servers or media servers.
In such cases, you may even want to create simple dedicated web pages that can be used to test these components so as not to have to go through your whole application logic.
Advantages of WebRTC test automation
Testing WebRTC applications is different from testing other web applications:
- WebRTC requires considerably more resources on the client and server side than most web applications
- A WebRTC application is more sensitive and susceptible to network conditions and available device resources
- In most WebRTC applications, there needs to be high synchronization of a test scenario across multiple devices and browsers
- Metrics required to understand if a test is successful or not are very different in WebRTC than in other web applications
Relying on manual testing means that a lot of the predictability and observability of the tests conducted is lost. Automation of WebRTC tests, if done correctly and with the appropriate tools can greatly enhance the stability of your application.
Designing WebRTC tests that scale
There are several things you need to take care of when designing your application for WebRTC tests that scale. If you don’t do them early on, then adding them later can be challenging.
Here are some best practices for designing scale into your WebRTC automation testing:
- Have the ability to configure test accounts for mock users with no captcha, OTP or 2FA on them. These are challenging to automate
- Create scripts that can generate as many pre-configured test accounts as needed. Doing this manually is going to be daunting, time consuming and error prone
- Prepare in advance an option to reduce the number of steps required in the UI to reach the media testing itself. Scheduling, form filling, and excessive selections and button clicking can all increase false positives in large stress tests for WebRTC
- Design your test script to scale in 2 axis: number of total participants and number of participants in a single session. This will give you more flexibility in your testing
- Have the ability to define different CPS (Calls Per Second) values, so that you can throttle and pace the entry of participants in large tests to fit your requirements
- Make it easy to run stress tests. If it takes you more than a few minutes to prepare and start running for a large stress test then you are doing it wrong
- Collect the metrics important to you from the test. You want to be able to understand the results quickly and be able to easily compare them with past test runs
testRTC service and how we can help
testRTC offers a self service platform for conducting WebRTC testing.
Our clients use us in a myriad of ways, anywhere from assisting them in manual testing, through continuous integration & regression testing up to large scale stress testing.
At the heart of the WebRTC testing service of testRTC there are 3 main technologies:
- Geographically spread browser automation at scale
- WebRTC metrics collection capabilities
- Visual and programmable analysis of WebRTC test results
We are continuously investing in each of these areas to make sure our clients can efficiently create, run and analyse their test scenarios in the quickest and simplest way.
You can create an evaluation account and start testing your WebRTC application now:
Take your WebRTC testing seriously
WebRTC has grown in recent years from a niche technology to an important piece of the fabric of the internet and of our communication toolkit. If you are developing products with WebRTC, it is paramount that you treat it like the first class citizen that it is in your testing as well.