We are asked from time to time if our platform can assist in catching lip sync issues, which are also known as A/V drift or other similar names.
The problem with lip sync testing is that it requires knowing the source media and being able to record, correlate and match it on the destination side. This is quite challenging with a generic platform like testRTC, especially when our control over what gets received is quite minimal:
- The browsers themselves don’t match audio and video in the source properly with fake media. This causes an inherent lip sync issues with our own source media in some scenarios. This is because audio and video are “faked” separately, so if one is opened prior to the other they won’t be synchronized and even if they were, the browser may not view them as starting at the exact same point time
- testRTC has no knowledge or understanding what the incoming media is and what source media this needs to be correlated with
- Network conditions affect incoming media, changing its resolution, frame rate, etc.
- Your application may process the media on the server side, changing it from the source and interrupting with lip syncing it programmatically
Due to the reasons above, we aren’t directly handling lip synchronization issues at testRTC. That said, there are several reasons for lip sync issues and A/V drifts to occur in WebRTC, which can be caught with testRTC: For the most part, A/V drift will occur not because the servers are “ruining” lip sync but rather because the resources on the client side can’t handle the load, so they throttle the incoming media and aren’t able to time the video with the voice.
To catch this on the client side, you can check certain elements in the testRTC results:
- Look for the CPU load in the performance tab. Try keeping it below 90% at all times
- The machines we provide operate on 2 cores by default, so if you are conducting a large video call, then you may hit CPU limits
- In such cases, you can increase the core counts of your machines (ask our support for assistance)
- Just be aware that many of your real users will have a similar experience, so our suggestion is to optimize for that type of machines as well
- Check the jitter and latency
- High jitter will result in bigger jitter buffers or lost packets, which may lead to loss of synchronization
- High RTT (latency) may cause browsers to play back audio instead of waiting for the video to arrive
- Look at the advanced WebRTC analytics
- If the frames decoded is consistently lower than the frames received, then you may have a performance issue to deal with
- If the time to decode a frame is higher than your frame rate (lets say 40 milliseconds on a 30 fps stream), then you will have a video drift issue