Unbiased Experiments in Congested Networks
This addresses a critical issue for network engineers and researchers by exposing flaws in standard evaluation practices, though it is incremental as it adapts existing designs from online platforms.
The paper tackles the problem of bias in A/B tests for networking algorithms due to congestion, showing that this bias can cause large errors, such as throughput appearing 150% higher or 75% lower in lab tests, and can lead to incorrect conclusions in real-world experiments like those with Netflix.
When developing a new networking algorithm, it is established practice to run a randomized experiment, or A/B test, to evaluate its performance. In an A/B test, traffic is randomly allocated between a treatment group, which uses the new algorithm, and a control group, which uses the existing algorithm. However, because networks are congested, both treatment and control traffic compete against each other for resources in a way that biases the outcome of these tests. This bias can have a surprisingly large effect; for example, in lab A/B tests with two widely used congestion control algorithms, the treatment appeared to deliver 150% higher throughput when used by a few flows, and 75% lower throughput when used by most flows-despite the fact that the two algorithms have identical throughput when used by all traffic. Beyond the lab, we show that A/B tests can also be biased at scale. In an experiment run in cooperation with Netflix, estimates from A/B tests mistake the direction of change of some metrics, miss changes in other metrics, and overestimate the size of effects. We propose alternative experiment designs, previously used in online platforms, to more accurately evaluate new algorithms and allow experimenters to better understand the impact of congestion on their tests.