Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach
This addresses a specific problem in recommendation systems for improving A/B test reliability, but it is incremental as it builds on existing methods to handle data distribution shifts.
The paper tackles interference in A/B tests caused by data training loops in recommendation systems by introducing a weighted training approach, which reduces bias and variance compared to other methods, as shown in simulation studies.
In modern recommendation systems, the standard pipeline involves training machine learning models on historical data to predict user behaviors and improve recommendations continuously. However, these data training loops can introduce interference in A/B tests, where data generated by control and treatment algorithms, potentially with different distributions, are combined. To address these challenges, we introduce a novel approach called weighted training. This approach entails training a model to predict the probability of each data point appearing in either the treatment or control data and subsequently applying weighted losses during model training. We demonstrate that this approach achieves the least variance among all estimators that do not cause shifts in the training distributions. Through simulation studies, we demonstrate the lower bias and variance of our approach compared to other methods.