Category Archives for "Testimonials"

2 How Talkdesk support solves customer network issues faster with testRTC

“The adoption of testRTC Network Testing at Talkdesk was really high and positive”

Earlier this month, I sat down with João Gaspar, Global Director, Customer Service at Talkdesk to understand more how they are using the new testRTC Network Testing product. This is the first time they’ve introduced a product that is designed for support teams, so this was an interesting conversation for me.

Talkdesk is the fastest growing cloud contact center solution today. They have over 1,800 customers across more than 50 countries. João oversees the global support team at Talkdesk with the responsibility to ensure clients are happy by offering proactive and transparent support.

All of Talkdesk customers make use of WebRTC as part of their call center capabilities. When call center agents open the Talkdesk application, they can receive incoming calls or dial outgoing calls directly from their browser, making use of WebRTC.

WebRTC challenges for cloud contact centers

The main challenge with cloud communication in contact centers is finding the reason for user complaints about call quality. Troubleshooting such scenarios to get to the root cause is very hard, and in almost all cases, Talkdesk has found out that it is not because of its communication infrastructure but rather due to issues between the customer’s agent and his firewall/proxy.

Issues vary from available bandwidth and quality in their internet connection, problems with their headphones, the machine they are using and a slew of other areas.

Talkdesk’s perspective and proactive focus to support means they’re engaging with clients not only when there are issues but through the entire cycle. For larger, enterprise deals,Talkdesk makes network assessments and provides recommendations to the client’s network team during the POC itself, not waiting for quality issues to crop later on in the process.

To that end, Talkdesk used a set of multiple tools, some of them running only on Internet Explorer and others testing network conditions but not necessarily focused on VoIP or Talkdesk’s communication infrastructure. It wasn’t a user friendly approach neither to Talkdesk’s support teams nor to the client’s agents and network team.

Talkdesk wanted a tool that provides quick analysis in a simple and accurate manner.

Adopting testRTC’s Network Testing product

Talkdesk decommissioned its existing analysis tools, preferring to use testRTC’s Network Testing product instead. WFor the client, with a click of a button, the clienthe is now able to  provides detailed analysis results to the Talkdesk support team within a minute. This enables faster response times and less frustration to Talkdesk and Talkdesk’s customer.

Today, all of the Talkdesk teams on the field, including support, networks and sales teams, make use of the testRTC Network Testing service. When a Talkdesk representative at a client location or remotely needs to understand the client’s network behavior, they send a link to a client, asking them to click the start button. testRTC Network Testing then conducts a set of network checks, immediately making the results to Talkdesk’s support.

testRTC’s backend dashboard for Talkdesk

The adoption of this product in Talkdesk was really high and positive. This is due to the simplicity and ease of use of it. For the teams on the field, this enables to easily engage with potential clients who haven’t signed a contract yet while investing very little resources.

The big win: turnaround time

testRTC’s Network Testing service doesn’t solve the client’s problems. There is no silver bullet there. Talkdesk support still needs to analyze the results, figure out the issues and work with the client on them.

testRTC’s Network Testing service enables Talkdesk to quickly understand if there are any blocking issues for clients and start engaging with clients sooner in the process. This dramatically reduces the turnaround time when issues are found, increasing transparency and keeping clients happier throughout the process.

Talkdesk Network Test service in action

On selecting testRTC

When Talkdesk searched for an alternative to their existing solution, they came to testRTC. They knew testRTC’s CEO through webinars and WebRTC related posts he published independently and via testRTC, and wanted to see if they can engage with testRTC on such a solution.

“testRTC’s Network Testing service reduces the turnaround time for us in understanding and addressing potential network issues with clients”

testRTC made a strategic decision to create a new service offering for WebRTC support teams, working closely with Talkdesk on defining the requirements and developing the service.

Throughout the engagement, Talkdesk found testRTC to be very responsive and pragmatic, making the adjustments required by Talkdesk during and after the initial design and development stages.

What brought confidence to Talkdesk is the stance that testRTC took in the engagement, making it clear that for testRTC this is a partnership and not a one-off service. For Talkdesk, this was one of the most important aspects.

How Nexmo Integrated testRTC into their Test Automation for the Nexmo Voice API

Nexmo found in testRTC a solution to solve its end-to-end media testing challenges for their Nexmo Voice API product, connecting PSTN to WebRTC and vice versa.

Nexmo is one of the top CPaaS vendors out there providing cloud communication APIs to developers, enabling enterprises to add communication capabilities into their products and applications.

One of Nexmo’s capabilities involves connecting voice calls between regular phone numbers (PSTN) to browsers (using WebRTC) and vice versa. This capability is part of the Nexmo Voice API.

Testing @ Nexmo

Catering to so many customers with ongoing deployments to production means that Nexmo needs to take testing seriously. One of the things Nexmo did early on was introduce automated testing, using the pytest framework. Part of this automated testing includes a set of regression tests –  a huge amount of tests that provide very high test coverage. Regression tests get executed whenever the Nexmo team has a new version to release, but these tests can also be launched “on demand” by any engineer, they can also be triggered by the Jenkins CI pipeline upon a merge to a particular branch.

At Nexmo, development teams are in charge of the quality of their code, so there is no separate QA team.

In many cases, launching these regression tests first creates a new environment, where the Nexmo infrastructure is launched dynamically on cloud servers. This enables developers to run multiple test sessions in parallel, each in front of their own sandboxed environment, running a different version of the service.

When WebRTC was added to Nexmo Voice API, there was a need to extend the testing environment to include support for browsers and for WebRTC technology.

On Selecting testRTC

“When it comes to debugging, when something has gone wrong, testRTC is the first place we’d go look. There’s a lot of information there”

Jamie Chapman, Voice API Engineer at Nexmo

Nexmo needed WebRTC end-to-end tests as part of their regression test suite for the Nexmo Voice API platform. These end-to-end tests were around two main scenarios:

  1. Dialing a call from PSTN and answering it inside a browser using WebRTC
  2. Calling a PSTN number directly from a browser using WebRTC

In both cases, their client side SDKs get loaded by a web page and tested as part of the scenario.

Nexmo ended up using testRTC as their tool of choice because it got the job done and it was possible to integrate it into their existing testing framework:

  • The python script used to define and execute a test scenario used testRTC’s API to dynamically create a test and run it on the testRTC platform
  • Environment variables specific to the dynamically created test environment got injected into the test
  • testRTC’s test result was then returned back to the python script to be recorded as part of the test execution result

This approach allowed Nexmo to integrate testRTC right into their current testing environment and test scripts.

Catering for Teams

The Voice API engineering team is a large oneAll these users have access to testRTC and they are able to launch regression tests that end up running testRTC scripts as well as using the testRTC dashboard to debug issues that are found.

The ability to have multiple users, each with their own credentials, running tests on demand when needed enabled increased productivity without dealing with coordination issues across the team members. The test results themselves get hosted on a single repository, accessible to the whole team, so all developers  can easily share faulty test results with the team .

Debugging WebRTC Issues

Nexmo has got regression testing for WebRTC off the ground by using testRTC. It does so by integrating with the testRTC APIs, scheduling and launching tests on demand from Nexmo’s own test environment. The tests today are geared towards providing end-to-end validation of media and connectivity between the PSTN network and WebRTC. Validation that testRTC takes care of by default.

When things break, developers check the results collected by testRTC. As Jamie Chapman, Voice API engineer at Nexmo said: “When it comes to debugging, when something has gone wrong, testRTC is the first place we’d go look. There’s a lot of information there”.

testRTC takes screenshots during the test run, as well as upon failure. It collects browser logs and webrtc-internals dump files, visualizing it all and making it available for debugging purposes. This makes testRTC a valuable tool in the development process at Nexmo.

On the Horizon

Nexmo is currently making use of the basic scripting capabilities of testRTC. It has invested in the API integration, but there is more that can be done.

Nexmo are planning to increase their use of testRTC in several ways in the near future:

Monitoring Vidyo’s WebRTC Infrastructure End-to-End on a Global Scale

Vidyo has been using testRTC for the past two years to monitor its global WebRTC infrastructure end-to-end.

Vidyo offers high quality cloud video conferencing services to its impressive list of customers. There are three main product lines at Vidyo:

  1. VidyoConnect – a managed enterprise meeting solution for team collaboration
  2. VidyoEngage – a live video chat platform for call center customer engagement
  3. Vidyo.io – cloud APIs for embedded video communications in applications

All of these product lines share the same core video platform with WebRTC capabilities.

Vidyo caters large enterprises in mission critical systems, so from the start, it put in place a sophisticated system to monitor its infrastructure and service. That system is built on top of Splunk, where logs from across its system gets aggregated and filtered, letting different types of alerts to bubble up to the relevant teams within Vidyo via PagerDuty or email, depending on the seriousness of the alert.

End-to-End Monitoring

Early on, Vidyo saw the need for an end-to-end monitoring capability within their monitoring system. A way that will simulate real customers from all over the globe and alert of any issues. This is why Vidyo selected to use testRTC.

testRTC enabled Vidyo to create a scenario where testRTC’s probes join calls on any of Vidyo’s cloud products, authenticate with the service, join a meeting room, send and receive voice and video data in real time.

While Vidyo monitored its different machines and subsystems already, adding testRTC meant it was capable of monitoring the service as experienced by real users, doing it with predictability over the scenario used and at scale.

Integrating with an existing monitoring system

Vidyo wanted to collect and push monitor run results from testRTC into its Splunk big data repository of machine data. Run results from testRTC are automatically inserted into Vidyo’s Splunk repository using testRTC’s webhook mechanism.

Collecting that data gave Vidyo the power to finetune the feedback it received from testRTC, deciding if a failure is of a low priority, occurring randomly or of high priority, such as a failure occurring across monitors in a short period of time.

A global infrastructure

Every data center that Vidyo operates from gets its own special treatment. For each of the product lines hosted within that data center, Vidyo has a running testRTC monitor for.

Each monitor makes use of probes running independently from different locations worldwide, which adds another layer of monitoring to the solution – testRTC is capable of checking different routes and behaviors, with the intent to catch network issues as early as possible as well.

Whenever a new data center opens up, or a new geography needs to be served, Vidyo is able to modify an existing monitor or create a new testRTC monitor to cover that location.

It just works

testRTC runs continuously and relentlessly, connecting calls via Vidyo’s platform. It does so in a predictable fashion, collecting all logs along the way. Vidyo have learned to see the value in such an approach – random failures can be debugged in post mortem, finding their root causes and assisting in finding bugs and points of failures in the system.

“testRTC is a key component in Vidyo’s monitoring system. Digging down to the root cause is part of the work culture at Vidyo, and using testRTC we have eyes on the system 24×7 and can investigate issues thoroughly ensuring operational excellence for the benefit of our customers. ”

Nahum Cohen, SVP, Service and Operations @ Vidyo

Using testRTC, Vidyo are able to find issues with data centers, networks and their platform before customers notice it, giving them the needed time to resolve these issues.

Moving Forward with testRTC

Vidyo is in the process of introducing testRTC’s monitors to additional data centers it is currently operating, making sure its service is monitored end-to-end for all of its locations.

How Houseparty uses testRTC as an integral part of its WebRTC testing

Houseparty selected testRTC for its WebRTC infrastructure regression testing through continuous integration.

Houseparty is a mobile group video chat application, where groups of up to eight friends gather to chat in virtual rooms. With over half a billion video chats conducted using WebRTC, Houseparty is massive in its scale. What makes Houseparty interesting, is that the majority of its users base are 24 old or younger audiences, spending upwards of 50 minutes a day inside the app.

Being a social platform, Houseparty has to innovate on a daily basis. This calls for frequent updates of its mobile applications and infrastructure. An update to the Houseparty backend infrastructure happens on a daily basis and the mobile apps are updated every two weeks on average.

In Search of a Regression Testing Tool

The developers at Houseparty wanted to get a kind of an early warning system in place. One that would tell the team if the changes being made are breaking the service for its users. And breakage here means a reduction in media quality or the inability to work in certain network conditions. What Houseparty’s developers were looking for was higher confidence in their version rollouts.

Houseparty already had stress testing capabilities in place, along with the ability to test their mobile applications. What they were missing was regression testing for the infrastructure. When a decision had to be made, Houseparty preferred to use testRTC’s testing service instead of building their own testing environment, saving months of effort of experienced WebRTC developers with the understanding that the end result would be inferior in terms of its feature set and capabilities.

By selecting testRTC, Houseparty’s developers  were able to improve their confidence level when upgrading the service for their millions of users.

“testRTC offered us the fastest and cheapest way to get the type of regression testing we needed, increasing the confidence we had when rolling out new releases of the Houseparty application”

Simplicity is Key

One of the key reasons for selecting testRTC was the simplicity of the service. From writing tests, through selecting the machines’ configuration and defining test success criteria down to integrating with the API.

The ability to pick different network configurations was really important to Houseparty. Using both the preconfigured settings as well as dynamically modifying network conditions enabled Houseparty to quickly and efficiently understand how the behavior of their application is affected.

Furthermore, by using test expectations in testRTC, a mechanism that lets developers set success and failure criteria for a test, based on metrics collected, Houseparty developers are alerted when results needs to be further analysed. This enables Houseparty’s developers to spend more time on their application and less in drilling down to results, trying to understand their meaning.

When drilling down to results is needed, then the graphs displayed assist the developers in debugging the problems and resolving them faster.

Outgoing and incoming video bitrate for an 8 people room with simulcast enabled

Mobile Only and WebRTC

While running predominantly as a mobile application, Houseparty’s video processing makes use of WebRTC. Houseparty is making a distinction between its application testing and infrastructure testing. It had in its arsenal existing tools that are being used to test its mobile clients. What it was looking for was a way to test their video infrastructure – their media servers and TURN servers – making sure they work as expected.

To that end, Houseparty is using a simple HTML page that can be used to create calls on its staging environment for the application. testRTC is then used to access that page and automate the testing process, simulating different network conditions while testing Houseparty’s video infrastructure.

Continuous Integration as a First Priority

Houseparty made the decision early on to use testRTC as part of their continuous integration environment. Using testRTC’s APIs, the developers at Houseparty were able to quickly integrate the testing scripts they’ve written in testRTC to their Jenkins automation server.

This allows Houseparty to run the testRTC regression tests every night. Integrating testRTC with Jenkins means that when tests complete, their results are reported back to Jenkins and from there they get sent to Slack, where developers get notified on potential failures.

Running testRTC tests nightly from Jenkins with integrated reporting and notifications

Moving Forward with testRTC

For Houseparty, the work is not done yet. testRTC is used on a daily basis, running a battery of tests designed to check their infrastructure. There are additional tests that are planned to be added to this test suite.

Peer-to-peer testing and direct TURN server testing will be added in the near future, increasing the coverage of regression testing done over testRTC.

How Clique Migrated Smoothly to the Newest AWS EC2 C5 Instance

In a need to focus resources on core activities, Clique Communications turned to testRTC for stress testing and sizing.

Clique API provides web-based voice and text application programming interfaces. In their eight years of existence, they have grown to support over 20 million users across 150 countries. This amounts to over 500 million minutes per month. Clique’s cloud services deliver multi-party voice that can be embedded by enterprises  into their own business processes.

What are their current goals?

  • Grow the business
  • Add features to improve customer service and experience
  • Offer value-added services

Adding WebRTC

Clique started working with WebRTC some 18 months ago with customers starting to use it at the end of 2017.

Today, Clique supports all major browsers – Chrome, Firefox, Safari, and Edge; enabling its customers to offer uninterrupted interactions with their users. When a user joins a conference, he/she can do so over PSTN, directly from the browser or from within a native application that utilizes Clique SDKs.

Making Use of testRTC

As with any other software product, Clique had to test and validate its solution. To that end, Clique had already been using  tools for handling call volumes and regression, testing the application and the SDKs. The challenge was the issue of scalability and quality of service, which is essential  when it comes to WebRTC support. Clique had a decision to make – either invest in building their own set of testing tools on top of open source frameworks such as Selenium, or opt for a commercial alternative. They decided to go with the latter and use testRTC. They also preferred using a third party tool for testing as they didn’t want to burden their engineering team.

Switching from AWS EC2 C4 to C5

Clique had previously used a standard instance on Amazon from the AWS EC2 C4 series but when the AWS EC2 C5 series came out, they wanted to take advantage of it – not only was it more economical but it also had better performance. Furthermore, knowing Amazon would release newer sets of servers that would need to be tested again, Clique required this process to be repeatable.

The Action Plan

Since Clique is an embeddable service, they decided it was most strategic to have a third party develop an application using the Clique client SDK and APIs, and use that application as a test framework that could scale and grow the performance of the platform. It was a wonderful opportunity to optimize their own resources and save on the instances that they deploy on Amazon. An added bonus was having a third party that could then be used by Clique’s customers and partners who are building applications hey can use as part of their development process.

Clique wrote  their own test scripts in testRTC. The main test scenario for Clique was having a moderator who creates the conference and then generates a URL for other participants in the conference to join. Once they figured out how to do that with testRTC, the rest was a piece of cake.

Using testRTC to assist in sizing the instances on the AWS has ancillary benefits beyond Clique’s core objectives. Clique tested the full life-cycle of its solution. From developing yet another application with its SDKs, integrating its APIs, to continuous integration & devops, Clique discovered  bugs that were then fixed, optimized performance and gave Clique confidence to run services at scale in next generation architectures.

“testRTC provided Clique with a reliable and repeatable mechanism to measure our CPaaS performance… allowing Clique to save money, remain confident in our architectural choices and more importantly showcase our platform to customers with the integrity of an independent test system.”

Moving Forward – Continued use of testRTC

There are a lot of moving pieces in Clique’s solution:  infrastructure in the backend, media servers , the WebRTC gateways. Features such as recording can fit into various components within the architecture, and Clique is always looking for ways to optimize and simplify.

testRTC helps Clique evaluate if the assumptions made in their architecture are valid by determining bottlenecks and identifying places of consolidation.

In the future, Clique will be looking at testRTC’s monitoring capability as well as using testRTC to instantiate browsers in different locations.