Instance-Conditional Timescales of Decay for Non-Stationary Learning
This addresses the challenge of non-stationary learning for practical ML systems, offering a novel method to handle concept drift without losing valuable past information.
The paper tackles the problem of slow concept drift in machine learning by proposing an optimization-driven approach to balance instance importance over large training windows, achieving up to 15% relative accuracy gains on a dataset of 39M photos over 9 years and outperforming state-of-the-art methods in continual learning settings.
Slow concept drift is a ubiquitous, yet under-studied problem in practical machine learning systems. In such settings, although recent data is more indicative of future data, naively prioritizing recent instances runs the risk of losing valuable information from the past. We propose an optimization-driven approach towards balancing instance importance over large training windows. First, we model instance relevance using a mixture of multiple timescales of decay, allowing us to capture rich temporal trends. Second, we learn an auxiliary scorer model that recovers the appropriate mixture of timescales as a function of the instance itself. Finally, we propose a nested optimization objective for learning the scorer, by which it maximizes forward transfer for the learned model. Experiments on a large real-world dataset of 39M photos over a 9 year period show upto 15% relative gains in accuracy compared to other robust learning baselines. We replicate our gains on two collections of real-world datasets for non-stationary learning, and extend our work to continual learning settings where, too, we beat SOTA methods by large margins.