All Posts by Tsahi Levent-Levi

0 Account information in qualityRTC

qualityRTC enables the use of client specific accounts.

By default, users can enter their email and reason. There is a possibility of adding additional custom fields for the user to fill out, but there is also an option to add an account field.

The account field can only be populated via the ?account= URL variable.

Once populated, it will appear in the backend dashboard for you to filter and review.

This enables easy grouping and identification of certain network tests to specific accounts and organizations.

If you need us to enable support for the account field in your project, let us know.

0 Best effort mechanism in stress testing

Why a best effort mode?

When conducting large stress tests, there is always a possibility that the number of probes you have asked to be allocated will not be granted. This isn’t due to testRTC directly, but rather due to the cloud vendors we use: The larger the number of probes we need to dynamically allocate in the cloud, the more the probability that some of these probe allocations will fail due to the cloud vendor’s own reasons.

This can be frustrating and wasteful when it comes to resource planning, and in many cases, what you actually want is to get as close as possible to that target number – allocating and running 996 probes instead of 1,000 would be a reasonable compromise for most.

When will best effort be invoked?

For that reason, from a certain test size (50 probes or more), testRTC will switch to using its Best Effort mode. This means that tests will run even if some of the probes fail to allocate for whatever reason.

In best effort mode, you will only be charged on probes that were actually allocated and used for the test run.

One important aspect here is that if a certain threshold of probes cannot be allocated, we will still fail the test and not run it – assuming that if you want a test with 1,000 probes and only 50 could be allocated – you’re not interested in it any more. That threshold is set to 5% by default and is configurable.

How to manually set best effort mode on and off?

You can use #probe-failure-mode run option, with the following values:

The failure threshold of best effort is set to 5% and can be configured using #best-effort-threshold run option.

0 WebRTC Non-Functional Testing Best Practices

As a company designed and built around WebRTC, we understand both VoIP and Web worlds, and the challenges and opportunities that they bring. Our insights, together with inputs received from leading WebRTC solutions providers and developers were translated into WebRTC expertise and testing best practices.

Our testRTC solution was designed and implemented with the goal to help you to test, operate, and optimize your WebRTC applications by following the described testing best practices.

If you decide to use testRTC or not is something you need to decide on your own. That said, here are tests you should conduct on your WebRTC product. It is the baseline that we suggest our customers focus on first with their products.

This best practices guide covers the following test areas:

1. Basic Sizing and Scaling

Here the focus is to test different types of call and situations:

1. Peer to Peer, typically 2 participant call running for longer duration – 20 min / 40 min / 60 min / 90 min
2. Group calls with a different number of participant
3. Group call running for longer duration – 20 min / 40 min / 60 min / 90 min
4. Participants joining, ending and rejoining from ongoing call
5. New participants joining and old participants ending the call
6. Find Max Session Size: the number of participants that can fit into a single session together
7. Check metrics (bitrates, packet loss and general stability) versus smaller session sizes
8. Find where metrics start to deteriorate and when metrics are unacceptable

2. Media Testing

The Idea is to used different media types and check how the bitrates and other metrics are handled:

1. Use 1080p inputs and see how it affects session bitrate and server-side load
2. Use 720p and VGA inputs and see if bitrate gets lowered considerably (WebRTC has a tendency of over-using bandwidth in lower resolutions by default)
3. Work in different simulated networks and see how it affects metrics

Note that the media files of testRTC come in different resolutions. The browser later scales down this raw input media to the resolutions needed for its own scenario.

3. Network Testing

1. Check Calls in different network conditions and combinations: look for how bitrates are maintained and changed to accommodate different bandwidth availability.
2. Combine 3-1 and 2
3. Simulate mid-call High packet loss or High Latency conditions
4. Simulate Call Drops for different users and check how the recovery takes place
5. Simulate Bandwidth change in the call and check how the service handles these. E.g. From Wifi to 4G to 3G to 2.5 G
6. Above should be done for the different type of users and how the situation is handled need to be reviewed

See rtcSetNetwork() and rtcSetNetworkProfile() on how to make these changes dynamically.

4. Firewall Testing

WebRTC peer to peer mechanism change when firewall comes into play.

The service settings on both client side, signaling and media servers need to be checked for their compatibility with different firewall profiles:

1. Check calls when blocking UDP (and later TCP) traffic
2. Check when only one participant blocks UDP (and later TCP) traffic
3. Check stress on TURN server by forcing TCP and running large scale sessions

5. Stress Testing

1. Find Max Users: Find the maximum number of users you can fit into your service, a single machine, a single data center
   • Use the smallest session sizes that make sense. We are here to determine theoretical maximum capacity
   • Check metrics (bitrates, packet loss and general stability) versus smaller number of total users
   • Focus on transition points of scale out and load balancing to see their impact
2. Max Everything: Find the maximum number of large sessions you can fit into your service
   • Use the largest session sizes that make sense, then grow the number of concurrent sessions
   • Check metrics (bitrates, packet loss and general stability) versus smaller number of users and sessions
   • Focus on transition points of scale out and load balancing to see their impact
3. Max Popular: Find the maximum number of popular sessions you can fit into your service
   • Use the average / popular session sizes that make sense, then grow the number of concurrent sessions
   • Check metrics (bitrates, packet loss and general stability) versus smaller number of users and sessions
   • Focus on transition points of scale out and load balancing to see their impact
4. Max Mixed: Mix the session sizes to meet the profile of your service
   • Use different session sizes in a single run, to test scenarios closer to the real world
   • Check metrics (bitrates, packet loss and general stability) versus smaller number of users and sessions
   • Focus on service stability
5. Load Conditions: Load the system and see what happens to new sessions that are added
   • Get the system loaded o 75%-90% of its capacity (you already found that in steps #2-#5)
   • Now add another session or two
   • Check metrics, including call setup time
6. Concurrency: Have multiple users join multiple sessions at the same time
   • Check for any race conditions and deadlocks, blocking users from joining successfully
   • Measure call setup time and success rate
7. Long Session: Run a long 1-2 hour session
   • Measure your backend memory for any leaks or other forms of thrashing
   • Join manually using your own device every once in awhile to monitor the progress manually
8. Improvisation: Change different conditions to see how your system fairs
   • Change network conditions (drop it 10 seconds), to see the effect it has on quality over time
   • Use different video resolutions as inputs to see if they strain your service differently
   • Work with different network configurations to validate your service robustness

For more details, please also read these checklists:

• Stress testing checklist
• Long running test checklist

6. Real world usage simulation using Dev Monitors

You can use upRTC monitoring service to complement your testing with these types of monitor configurations:

1. 1 – Max Session Size and 1 Popular Session Size Group sessions
   • Call length 1 hour – Frequency every 1.5 hours
2. 3 Popular Session Size group calls
   • Call length 10 min – Frequency every 15 min
   • These will have different networks like Wifi, DSL, 4G and 3G
3. 1 Popular Session Size group call
   • Call length 5 min – Frequency every 15 min
   • Simulate Call Drop
4. 1 Popular Session Size group call
   • Call length 5 min – Frequency every 15 min
   • Two users in Poor 4g or Poor 3G network for half of the call
5. 4 two person call
   • Call length 10 min – Frequency every 15 min
   • Users in variety of network conditions
6. 1 Popular Session Size group call
   • Call length 5 min – Frequency every 15 min
   • For two users Bandwidth shrinks – Wifi to 4G to 3G to 2.5 G

0 How would bad routing look like on qualityRTC?

One of the leading factors to quality issues with WebRTC is the distance between the user and your backend infrastructure. Even if there is a nearby server, there can be limiting factors such as the use of a VPN service by your customer or just bad routing rules, that get your user connected to the wrong server.

Here is how bad routing tend to look like in qualityRTC:

The WebRTC network test above was conducted in Israel, using a ProtonVPN client that was connected to a VPN server in Japan.

Here’s what we can see:

• VIDEO BANDWIDTH shows very high Round Trip Time (marked in red). This will result in poor quality and high latency for a video conversation
• VIDEO QUALITY and VIDEO P2P also show high Round Trip Time values
• VIDEO P2P shows low bitrate (it usually gets to 2Mbps)
• LOCATION shows the user is in Japan, while we know he is located elsewhere (in Israel)
• BANDWIDTH SPEED shows high jitter on uploading and downloading over HTTPS, with very poor connections of around 1.2Mbps
• TURN CONNECITIVTY shows slow connection times, but is still capable to connect UDP

What to do in such a case?

Ask your user if he is using a VPN. If he is, suggest removing it or configuring it in a way to allow WebRTC traffic to go unhindered for your service

0 Network testing for WebRTC video applications

qualityRTC is quite suitable for testing network conditions of WebRTC video applications. These can be web and video conferencing and collaboration applications, telehealth services, remote education applications, etc.

There are 3 test widgets uniquely developed to address WebRTC video specific issues:

1. VIDEO BANDWIDTH – for understanding general outgoing available video bandwidth
2. VIDEO QUALITY – for direct connectivity tests to media servers
3. VIDEO P2P – for testing peer-to-peer connection scenarios

These widgets can be customized to work directly with your video backend infrastructure.

Here is our qualityRTC demo in action for a video-based application:

0 VIDEO BANDWIDTH shows low bitrate while BANDWIDTH SPEED shows high bitrates

In the screen shot below, a user comes with a result in qualityRTC that shows low VIDEO BANDWIDTH bitrates while BANDWIDTH SPEED is showing high bitrates.

Here is what we can glean out of the information:

1. TURN CONNECTIVITY shows that the user’s network is open to UDP traffic and round trip time is low (the time it takes to connect to the UDP and TCP TURN servers is below 500ms)
2. Both VIDEO BANDWIDTH and VIDEO QUALITY shows bitrate estimation of 29 kbps which is very low
3. VIDEO QUALITY shows bitrates of 32-37kbps for the actual video sent and received, which is very low
4. BANDWIDTH SPEED shows high bitrates of 24-26Mbps on the uplink and the downlink

BANDWIDTH SPEED refers to HTTPS traffic, taking place over TCP.

VIDEO BANDWIDTH and VIDEO QUALITY traffic takes place over UDP by default.

The reason for the difference might stem from throttling of UDP traffic that is done by the user’s firewall.

What to do in such a case?

1. Check if UDP is being throttled directly for that user
2. Check if the firewall is configured to block P2P encrypted traffic or bittorrent traffic. Such features are known to also block and throttle WebRTC traffic

0 rtcSetNetwork()

Change the network configuration during a test run dynamically to simulate changing network conditions. Also see rtcSetNetworkProfile().

Arguments

JSON object

The JSON object is defined as:

    client.rtcSetNetwork({
        outgoing: {
            packetloss: [%],
            jitter: [milliseconds],
            latency: [milliseconds],
            bitrate: [Kbps]
        },
        incoming: {
            packetloss: [%],
            jitter: [milliseconds],
            latency: [milliseconds],
            bitrate: [Kbps]
        }
    });
Code language: CSS (css)

Code examples

client.rtcSetNetwork({
    outgoing: {
        packetloss: 0.01,
        jitter: 0,
        latency: 0,
        bitrate: 5000
    },
    incoming: {
        packetloss: 0.01,
        jitter: 0,
        latency: 0,
        bitrate: 5000
    }
});
Code language: CSS (css)

The script above will change the network configuration to allow up to 5mbps of bitrate with 0.01% packet loss on both incoming and outgoing direction.

0 How to test local signaling servers with self signed certificate?

testRTC runs in the cloud by default. The probes we use that simulate live users come from the public internet. As such, if your WebRTC implementation is installed on a local network, on a machine or a lab, and you need it tested, then the way to go is to do the following:

Use ngrok (or similar)

You can configure a public server to act like a proxy that tunnels all signaling towards your local server.

One such popular service is ngrok.

Ignore certificate errors

Since you are using a local server, the certificate used is going to be self signed. Chrome doesn’t like these types of certificates, so you’ll need to tell Chrome to just ignore it.

For that purpose, add to the run option of the test the following line:

#chrome-cli:ignore-certificate-errorsCode language: CSS (css)

You can find a few more useful chrome debug settings here.

0 Supported infrastructure in qualityRTC

qualityRTC is designed to work with virtually any WebRTC infrastructure. In the initial setup and customization page of onboarding a new client to qualityRTC, we implement the necessary parts to get these integrations up and running.

Our team has been doing that for quite some time, and we have an extensive and growing library of backend WebRTC servers that we’ve already integrated with.

Here are some of the supported implementations we have (if you use them, then the setup part of onboarding to qualityRTC will be simpler):

• Agora Video SDK
• Avaya Client SDK
• AWS Chime SDK
• AudioCodes WebRTC Solutions for Enterprises
• Cisco Webex SDK
• coturn TURN server
• FreeSWITCH
• Intel Collaboration Suite
• Janus
• Jitsi
• jsSIP
• Licode
• LiveSwitch (Frozen Mountain)
• mediasoup
• OpenVidu
• Pexip
• Twilio Programmable Voice
• Twilio Programmable Video
• Vonage Video API
• Vonage In-App Voice
• XirSys