Offline to Online Learning for Real-Time Bandwidth Estimation
This addresses the problem of adapting bandwidth estimation to individual user network conditions for real-time video applications, representing an incremental improvement over existing methods.
The paper tackles the challenge of personalizing bandwidth estimation algorithms for real-time video applications by proposing Merlin, an imitation learning-based solution that transforms heuristic-based algorithms into neural networks for data-driven personalization. In real intercontinental videoconferencing tests, Merlin matches heuristic policy performance and achieves up to 7.8% QoE improvement after finetuning, while converging with 80% fewer samples than online RL methods.
Real-time video applications require accurate bandwidth estimation (BWE) to maintain user experience across varying network conditions. However, increasing network heterogeneity challenges general-purpose BWE algorithms, necessitating solutions that adapt to end-user environments. While widely adopted, heuristic-based methods are difficult to individualize without extensive domain expertise. Conversely, online reinforcement learning (RL) offers ease of customization but neglects prior domain expertise and suffers from sample inefficiency. Thus, we present Merlin, an imitation learning-based solution that replaces the manual parameter tuning of heuristic-based methods with data-driven updates to streamline end-user personalization. Our key insight is that transforming heuristic-based BWE algorithms into neural networks facilitates data-driven personalization. Merlin utilizes Behavioral Cloning to efficiently learn from offline telemetry logs, capturing heuristic policies without live network interactions. The cloned policy can then be seamlessly tailored to end user network conditions through online finetuning. In real intercontinental videoconferencing calls, Merlin matches our heuristic's policy with no statistically significant differences in user quality of experience (QoE). Finetuning Merlin's control policy to end-user environments enables QoE improvements of up to 7.8% compared to the heuristic policy. Lastly, our IL-based design performs competitively with current state-of-the-art online RL techniques but converges with 80% fewer videoconferencing samples, facilitating practical end-user personalization.