Efficient Hyperparameter Search for Non-Stationary Model Training
This addresses the problem of expensive model training for recommendation and advertising systems, though it appears incremental as it builds on existing hyperparameter optimization concepts.
The paper tackles the high cost of hyperparameter search for online learning systems with non-stationary data by introducing a two-stage paradigm that first identifies promising configurations and then trains only those candidates fully. The result is a reduction in total hyperparameter search cost by up to 10× on a public benchmark and significant efficiency gains in an industrial advertising system.
Online learning is the cornerstone of applications like recommendation and advertising systems, where models continuously adapt to shifting data distributions. Model training for such systems is remarkably expensive, a cost that multiplies during hyperparameter search. We introduce a two-stage paradigm to reduce this cost: (1) efficiently identifying the most promising configurations, and then (2) training only these selected candidates to their full potential. Our core insight is that focusing on accurate identification in the first stage, rather than achieving peak performance, allows for aggressive cost-saving measures. We develop novel data reduction and prediction strategies that specifically overcome the challenges of sequential, non-stationary data not addressed by conventional hyperparameter optimization. We validate our framework's effectiveness through a dual evaluation: first on the Criteo 1TB dataset, the largest suitable public benchmark, and second on an industrial advertising system operating at a scale two orders of magnitude larger. Our methods reduce the total hyperparameter search cost by up to 10$\times$ on the public benchmark and deliver significant, validated efficiency gains in the industrial setting.