Support & information center

As a company designed and built around WebRTC, we understand both VoIP and Web worlds, and the challenges and opportunities that they bring. Our insights, together with inputs received from leading WebRTC solutions providers and developers were translated into WebRTC expertise and testing best practices.

Our testRTC solution was designed and implemented with the goal to help you to test, operate, and optimize your WebRTC applications by following the described testing best practices.

If you decide to use testRTC or not is something you need to decide on your own. That said, here are tests you should conduct on your WebRTC product. It is the baseline that we suggest our customers focus on first with their products.

This best practices guide covers the following test areas:

1. Basic Sizing and Scaling

Here the focus is to test different types of call and situations:

1. Peer to Peer, typically 2 participant call running for longer duration – 20 min / 40 min / 60 min / 90 min
2. Group calls with a different number of participant
3. Group call running for longer duration – 20 min / 40 min / 60 min / 90 min
4. Participants joining, ending and rejoining from ongoing call
5. New participants joining and old participants ending the call
6. Find Max Session Size: the number of participants that can fit into a single session together
7. Check metrics (bitrates, packet loss and general stability) versus smaller session sizes
8. Find where metrics start to deteriorate and when metrics are unacceptable

2. Media Testing

The Idea is to used different media types and check how the bitrates and other metrics are handled:

1. Use 1080p inputs and see how it affects session bitrate and server-side load
2. Use 720p and VGA inputs and see if bitrate gets lowered considerably (WebRTC has a tendency of over-using bandwidth in lower resolutions by default)
3. Work in different simulated networks and see how it affects metrics

Note that the media files of testRTC come in different resolutions. The browser later scales down this raw input media to the resolutions needed for its own scenario.

3. Network Testing

1. Check Calls in different network conditions and combinations: look for how bitrates are maintained and changed to accommodate different bandwidth availability.
2. Combine 3-1 and 2
3. Simulate mid-call High packet loss or High Latency conditions
4. Simulate Call Drops for different users and check how the recovery takes place
5. Simulate Bandwidth change in the call and check how the service handles these. E.g. From Wifi to 4G to 3G to 2.5 G
6. Above should be done for the different type of users and how the situation is handled need to be reviewed

See rtcSetNetwork() and rtcSetNetworkProfile() on how to make these changes dynamically.

4. Firewall Testing

WebRTC peer to peer mechanism change when firewall comes into play.

The service settings on both client side, signaling and media servers need to be checked for their compatibility with different firewall profiles:

1. Check calls when blocking UDP (and later TCP) traffic
2. Check when only one participant blocks UDP (and later TCP) traffic
3. Check stress on TURN server by forcing TCP and running large scale sessions

5. Stress Testing

1. Find Max Users: Find the maximum number of users you can fit into your service, a single machine, a single data center
   • Use the smallest session sizes that make sense. We are here to determine theoretical maximum capacity
   • Check metrics (bitrates, packet loss and general stability) versus smaller number of total users
   • Focus on transition points of scale out and load balancing to see their impact
2. Max Everything: Find the maximum number of large sessions you can fit into your service
   • Use the largest session sizes that make sense, then grow the number of concurrent sessions
   • Check metrics (bitrates, packet loss and general stability) versus smaller number of users and sessions
   • Focus on transition points of scale out and load balancing to see their impact
3. Max Popular: Find the maximum number of popular sessions you can fit into your service
   • Use the average / popular session sizes that make sense, then grow the number of concurrent sessions
   • Check metrics (bitrates, packet loss and general stability) versus smaller number of users and sessions
   • Focus on transition points of scale out and load balancing to see their impact
4. Max Mixed: Mix the session sizes to meet the profile of your service
   • Use different session sizes in a single run, to test scenarios closer to the real world
   • Check metrics (bitrates, packet loss and general stability) versus smaller number of users and sessions
   • Focus on service stability
5. Load Conditions: Load the system and see what happens to new sessions that are added
   • Get the system loaded o 75%-90% of its capacity (you already found that in steps #2-#5)
   • Now add another session or two
   • Check metrics, including call setup time
6. Concurrency: Have multiple users join multiple sessions at the same time
   • Check for any race conditions and deadlocks, blocking users from joining successfully
   • Measure call setup time and success rate
7. Long Session: Run a long 1-2 hour session
   • Measure your backend memory for any leaks or other forms of thrashing
   • Join manually using your own device every once in awhile to monitor the progress manually
8. Improvisation: Change different conditions to see how your system fairs
   • Change network conditions (drop it 10 seconds), to see the effect it has on quality over time
   • Use different video resolutions as inputs to see if they strain your service differently
   • Work with different network configurations to validate your service robustness

For more details, please also read these checklists:

• Stress testing checklist
• Long running test checklist

6. Real world usage simulation using Dev Monitors

You can use upRTC monitoring service to complement your testing with these types of monitor configurations:

1. 1 – Max Session Size and 1 Popular Session Size Group sessions
   • Call length 1 hour – Frequency every 1.5 hours
2. 3 Popular Session Size group calls
   • Call length 10 min – Frequency every 15 min
   • These will have different networks like Wifi, DSL, 4G and 3G
3. 1 Popular Session Size group call
   • Call length 5 min – Frequency every 15 min
   • Simulate Call Drop
4. 1 Popular Session Size group call
   • Call length 5 min – Frequency every 15 min
   • Two users in Poor 4g or Poor 3G network for half of the call
5. 4 two person call
   • Call length 10 min – Frequency every 15 min
   • Users in variety of network conditions
6. 1 Popular Session Size group call
   • Call length 5 min – Frequency every 15 min
   • For two users Bandwidth shrinks – Wifi to 4G to 3G to 2.5 G