Tag Archives for " API integration "

Monitoring WebRTC applications using testRTC

This time, I want to do a quick recap on a webinar we hosted last week. It dealt with monitoring WebRTC applications, and as is usual, we took the approach of doing a demo.

For this one, I’ve prepared for over a month. I’ve created 3 separate monitors, running on AppRTC, appear.in and Jitsi Meet.

Why did I pick these 3 services?

  1. They are all public, and don’t throttle or cap you in how they work (a lot of demos out there do that)
  2. They are quite different from one another, yet similar at the same time (not sure if that means anything, but it feels that way)
  3. AppRTC is used by many to build their first application, and it is Google’s “hello world” application for WebRTC (it is also unstable, which is a lot of fun monitoring)
  4. appear.in and Jitsi are widely known and quite popular. They are also treated as real services (where AppRTC is merely a demo)

What was the scenario I used?

  1. Create a meeting URL (either ad-hoc or predefined)
  2. Join the URL
  3. Wait for 2 full minutes so we have data to chew on
  4. Run the above every 15 minutes for a period of a bit over a month

Connecting the dots

I wanted to show a bit more than what you see in the testRTC dashboard. Partly out of curiosity but also because many of our monitoring customers do just that – connect the results they get from testRTC’s monitor runs in their own applications.

So I took this approach:

I’ve created a Google Sheet to collect all the results.

In Zapier, I then created a Webhook that collected the results into that Google Sheet.

In testRTC, I’ve configured a webhook to connect to Zapier.

10,000 monitor runs later… there was enough data to chew on.

Initial insights

Here’s how the data looks like on the testRTC dashboard:

The first row? AppRTC. It fails every once in awhile for 2+ hours at a time.

Jitsi had a few hiccups. appear.in was almost flawless.

Here’s what happened when I created a quick pivot table of it on my Google Sheet:

AppRTC was down 6% of the time…

Media quality

We now have our own unique ranking system for WebRTC media quality in test results.

I’ve showed that during the webinar as well, checking out the various services fair with maintaining stable quality throughout the month. Interestingly, they behaved rather differently from one another.

Watch the webinar to learn more about it.

The webinar and demo

Most of the webinar was a long demo session. You can view it all here:

You can open up your own testRTC account and play with our service a bit under evaluation.

Our next webinar – CI/CD

Lots of the work our customers do involved automation. They write scripts to automate testing and then they want to automate running these scripts when needed. This usually means running a nightly build, on code check in, etc.

testRTC caters to that through a simple to use API, which is what I’ll be demoing in the next webinar. And I won’t be doing it alone – I’ll be joined by Gustavo Garcia of Houseparty. Gustavo was one of our first customers to make use of our APIs in such a way.

How Houseparty uses testRTC as an integral part of its WebRTC testing

Houseparty selected testRTC for its WebRTC infrastructure regression testing through continuous integration.

Houseparty is a mobile group video chat application, where groups of up to eight friends gather to chat in virtual rooms. With over half a billion video chats conducted using WebRTC, Houseparty is massive in its scale. What makes Houseparty interesting, is that the majority of its users base are 24 old or younger audiences, spending upwards of 50 minutes a day inside the app.

Being a social platform, Houseparty has to innovate on a daily basis. This calls for frequent updates of its mobile applications and infrastructure. An update to the Houseparty backend infrastructure happens on a daily basis and the mobile apps are updated every two weeks on average.

In Search of a Regression Testing Tool

The developers at Houseparty wanted to get a kind of an early warning system in place. One that would tell the team if the changes being made are breaking the service for its users. And breakage here means a reduction in media quality or the inability to work in certain network conditions. What Houseparty’s developers were looking for was higher confidence in their version rollouts.

Houseparty already had stress testing capabilities in place, along with the ability to test their mobile applications. What they were missing was regression testing for the infrastructure. When a decision had to be made, Houseparty preferred to use testRTC’s testing service instead of building their own testing environment, saving months of effort of experienced WebRTC developers with the understanding that the end result would be inferior in terms of its feature set and capabilities.

By selecting testRTC, Houseparty’s developers  were able to improve their confidence level when upgrading the service for their millions of users.

“testRTC offered us the fastest and cheapest way to get the type of regression testing we needed, increasing the confidence we had when rolling out new releases of the Houseparty application”

Simplicity is Key

One of the key reasons for selecting testRTC was the simplicity of the service. From writing tests, through selecting the machines’ configuration and defining test success criteria down to integrating with the API.

The ability to pick different network configurations was really important to Houseparty. Using both the preconfigured settings as well as dynamically modifying network conditions enabled Houseparty to quickly and efficiently understand how the behavior of their application is affected.

Furthermore, by using test expectations in testRTC, a mechanism that lets developers set success and failure criteria for a test, based on metrics collected, Houseparty developers are alerted when results needs to be further analysed. This enables Houseparty’s developers to spend more time on their application and less in drilling down to results, trying to understand their meaning.

When drilling down to results is needed, then the graphs displayed assist the developers in debugging the problems and resolving them faster.

Outgoing and incoming video bitrate for an 8 people room with simulcast enabled

Mobile Only and WebRTC

While running predominantly as a mobile application, Houseparty’s video processing makes use of WebRTC. Houseparty is making a distinction between its application testing and infrastructure testing. It had in its arsenal existing tools that are being used to test its mobile clients. What it was looking for was a way to test their video infrastructure – their media servers and TURN servers – making sure they work as expected.

To that end, Houseparty is using a simple HTML page that can be used to create calls on its staging environment for the application. testRTC is then used to access that page and automate the testing process, simulating different network conditions while testing Houseparty’s video infrastructure.

Continuous Integration as a First Priority

Houseparty made the decision early on to use testRTC as part of their continuous integration environment. Using testRTC’s APIs, the developers at Houseparty were able to quickly integrate the testing scripts they’ve written in testRTC to their Jenkins automation server.

This allows Houseparty to run the testRTC regression tests every night. Integrating testRTC with Jenkins means that when tests complete, their results are reported back to Jenkins and from there they get sent to Slack, where developers get notified on potential failures.

Running testRTC tests nightly from Jenkins with integrated reporting and notifications

Moving Forward with testRTC

For Houseparty, the work is not done yet. testRTC is used on a daily basis, running a battery of tests designed to check their infrastructure. There are additional tests that are planned to be added to this test suite.

Peer-to-peer testing and direct TURN server testing will be added in the near future, increasing the coverage of regression testing done over testRTC.