The most important decision we made was one that took place before testRTC became a company. It was the decision to use a web browser as the agent/probe for our service instead of building something on top of WebRTC directly or god forbid GStreamer.
Here are a few things we can do because we use real browsers in WebRTC testing:
#1 – Time to Market
Chrome just released version 49.
How long will it take for you to test its behavior against your WebRTC service if you are simulating traffic instead of using this browser directly?
For us, the moment a browser gets released is almost the moment we can enable it for our customers.
To top it off, we enable access to our customers to the upcoming browser versions as well – beta and unstable. This helps those who need to test their service check and get some confidence in their service when running them against future versions of browsers.
Even large players in the WebRTC industry can be hit by browser updates – TokBox did some time ago, so being able to test and validate such issues earlier on is imperative.
#2 – Pace of Change
VP9? H.264? ORTC APIs? Deprecation of previous APIs? Replacement of the echo canceler? Addition of local recording APIs? Media forwarding?
Every day in WebRTC brings with it yet another change.
Browsers get updated in 6-8 weeks cycles, and the browser vendors aren’t shy about removing features or adding new ones.
Maintaining such short release cycles is hellishly tough. For an established vendors (testing or otherwise), it is close to impossible – they are used to 6-12 months release cycles at best. For startups it is just too much of a hassle to run at these speeds – what you are trying to achieve at this point is leverage others and focus on the things you need to do.
So if the browser is there, it gets frequently updated, and it is how the end users end up running the service, why can’t we use it ourselves to leverage both automated and manual testing?
It was stupidly easy for me to test VP9 with testRTC even before it was officially released in the browser. All I had to do was pick the unstable version of the browser testRTC supports and… run the test script we already had.
The same is true for all other changes browsers make in WebRTC or elsewhere – they become available to us ad our customers immediately. And in most cases, with no development at all on our part.
#3 – Closest to Reality
You decided to use someone who simulates traffic and follows the WebRTC spec for your testing.
Great.
But does it act like a browser?
Chrome and Firefox act different through the API calls and look different on the wire. Hell – the same browser in two different versions acts differently.
Then why the hell use a third party who read the WebRTC spec and interpreted it slightly different than the browser used at the end of the day? Count the days. In each passing day, that third party is probably getting farther away from the browsers your customers are using (until someone takes the time and invests in updating it).
#4 – Signaling Protocols
When we started this adventure we are on with testRTC, we needed to decide what signaling to put on top of WebRTC.
Should it be SIP over WebSocket? Covering the traditional VoIP market.
Maybe we should to for XMPP. Over BOSH. Or Comet. Or WebSocket. Or not at all.
Should we add an API on top that the customer integrates with in order to simulate the traffic and connect to his own signaling?
All these alternatives had serious limitations:
- Picking a specific signaling protocol would have limited our market drastically
- Introducing an integration API for any signaling meant longer customer acquisition cycles and reducing our target market (yet again)
A browser on the other hand… that meant that whatever the customer decided to do – we immediately support. The browser is going to drive the interaction anyway. Which is why we ended up using browsers as the main focus of our WebRTC testing and monitoring service.
#5- Functional Testing and Business Processes
WebRTC isn’t tested in vacuum. When you used to use VoIP – things were relatively easy. You have the phone system. It is a service. You know what it does and how it works. You can test it and any of its devices and building blocks – it is all standardized anyway.
WebRTC isn’t like that. It made VoIP into a feature. You have a dating site. In that site people interact in multiple ways. They may also be doing voice and video calls. But how they reach out to each other, and what business processes there are along the way – all these aren’t related to VoIP at all.
Having a browser meant we can add these types of tests to our service. And we have customers who check the logic of their site and backend while also checking the media quality and the WebRTC traffic. It means there’s more testing you can do and more functionality of your own service you can cover with a single tool.
Thinking of Testing Your WebRTC Service?
Make sure a considerable part of the testing you do happens with the help of browsers.
Simulators and traffic generators are nice, but they just don’t cut it for this tech.
