Adversarial Validation Approach to Concept Drift Problem in User Targeting Automation Systems at Uber
This addresses performance degradation in user targeting systems for companies like Uber, though it appears incremental as it builds on existing adversarial validation methods.
The paper tackles concept drift in user targeting automation systems by introducing an adversarial validation approach that detects drift before inference and adapts predictions, showing effectiveness on AutoML3 challenge data and Uber's MaLTA system.
In user targeting automation systems, concept drift in input data is one of the main challenges. It deteriorates model performance on new data over time. Previous research on concept drift mostly proposed model retraining after observing performance decreases. However, this approach is suboptimal because the system fixes the problem only after suffering from poor performance on new data. Here, we introduce an adversarial validation approach to concept drift problems in user targeting automation systems. With our approach, the system detects concept drift in new data before making inference, trains a model, and produces predictions adapted to the new data. We show that our approach addresses concept drift effectively with the AutoML3 Lifelong Machine Learning challenge data as well as in Uber's internal user targeting automation system, MaLTA.