Why a best effort mode?
When conducting large stress tests, there is always a possibility that the number of probes you have asked to be allocated will not be granted. This isn’t due to testRTC directly, but rather due to the cloud vendors we use: The larger the number of probes we need to dynamically allocate in the cloud, the more the probability that some of these probe allocations will fail due to the cloud vendor’s own reasons.
This can be frustrating and wasteful when it comes to resource planning, and in many cases, what you actually want is to get as close as possible to that target number – allocating and running 996 probes instead of 1,000 would be a reasonable compromise for most.
When will best effort be invoked?
For that reason, from a certain test size (50 probes or more), testRTC will switch to using its Best Effort mode. This means that tests will run even if some of the probes fail to allocate for whatever reason.
In best effort mode, you will only be charged on probes that were actually allocated and used for the test run.
One important aspect here is that if a certain threshold of probes cannot be allocated, we will still fail the test and not run it – assuming that if you want a test with 1,000 probes and only 50 could be allocated – you’re not interested in it any more. That threshold is set to 5% by default and is configurable.
How to manually set best effort mode on and off?
You can use #probe-failure-mode run option, with the following values:
| Value | Description |
|---|---|
| auto | This is the default value of the best effor mode. Let testRTC decide automatically if it makes sense to use best effort mode or not: Below 50 probes in a test, the test runs only if all probes were successfully allocated Above 50 probes and with session size smaller than 10, the test runs in be-session mode (see below) Above 50 probes and with session size 10 or larger, the test runs in be-probe mode (see below) |
| exact | Force testRTC to run the test only if all probes were successfully allocated |
| be-probe | Allow testRTC to run tests even if not all the probes were allocated. Use all probes that were allocated in the test |
| be-session | Allow testRTC to run tests even if not all the probes were allocated. Make sure to use probes only if they fill out the sessions properly. For example, if the session size is configured to 4, and on best effort 497 probes out of 500 were allocated successfully, then only 496 will be used for the test. |
The failure threshold of best effort is set to 5% and can be configured using #best-effort-threshold run option.