By default, testRTC limits tests to up to 5 minutes in length. You can change that by using the #timeout run option. The longer the test, the more you’ll need to plan and prepare for running it successfully.
If you are trying to run long tests, also known as stamina tests, then you should read on.
We’ve compiled a set of checklist items for you to go through if you want to run tests longer than 15 minutes.
Table of contents
1. Don’t forget the #timeout
By default, tests are 5 minutes long. You’ll need to increase that by using the #timeout run option.
For an hour long test, go for ~80 minutes of timeout: #timeout:80
This should give you enough time to setup the session and tear it down in the end properly.
2. The way to 2 hours goes through 15 minutes
If you are planning on conducting a 2 hour long test, you should first use this checklist and prepare for a 15 minutes test, then move towards 40-60 minutes and only then go for the full 2 hours.
Why? So that you catch errors earlier on. This will use less minutes and less of your time to get the script ready for that big test.
3. Long test, stress test and only then a long stress test
Most are looking to run a long stress test, running 100 or more probes for an hour or more. This is great, but don’t go there immediately.
Do two things first:
- Focus on a small number of probes in a long test. 2-4 probes should be fine for this. It will allow you to weed out any issues in these tests and to use this checklist while at it
- Separately, run a short test with lots of probes. For this one, consult with our stress testing checklist
Once you’ve done the two preparations above, you should be ready and have the confidence needed to run a long stress test.
4. Pace the script ending
At the end of test you should have add some delay between the probes as they leave the test. This will help the cleanup, uploading and analysis processes that takes place by testRTC at that point in time – otherwise, they might just fail due to high load.
This is especially important in tests with long duration.
var agentNumber = process.env.RTC_AGENT_NUM;
client.pause(agentNumber * 500);Code language: JavaScript (javascript)Add the code snippet above at the end of your test script to add a 500 milliseconds of delay between each probe that is leaving the test.
For shorter tests, we’re fine with 100 milliseconds of a delay. For longer tests, there’s a lot more data being collected, and this needs to be accounted for. 2 seconds delay should be enough to shut down the session properly.
5. Metrics collection flags
Make sure you add #getstat #webrtc-internals:false flags in your run options.
webrtc-internals collects only the last 15 minutes of metrics so for long tests its rather useless (and collecting it means wasting yet more time). Better just use #webrtc-internals:false.
In case your test does not support #getstat – contact us.
6. Don’t fall for Selenium session expirations
Selenium sessions expire automatically because of inactivity which can cause tests to fail. For the most part, in an hour-long test, you are going to just wait for media to get sent or received.
Due to this, you need to let Selenium know things are alive and well. This can be done by taking some kind of an action on the screen every 10-15 minutes or so.
The easiest way to do that without disturbing the test is by moving the mouse.
client
.rtcScreenshot("In call")
.pause(TEST_DURATION * MIN / 4)
.moveTo(null, 110, 100)
.pause(TEST_DURATION * MIN / 4)
.rtcScreenshot("Mid call")
.pause(TEST_DURATION * MIN / 4)
.moveTo(null, 110, 10)
.pause(TEST_DURATION * MIN / 4)
.rtcScreenshot("End call");Code language: JavaScript (javascript)