Adaptive Regularization for Large-Scale Sparse Feature Embedding Models
This addresses a critical issue for practitioners in search, advertising, and recommendation domains, offering a practical solution to a known bottleneck.
The paper tackled the one-epoch overfitting problem in large-scale sparse feature embedding models used for CTR and CVR estimation, proposing an adaptive regularization method that prevents performance degradation in multi-epoch training and improves results within a single epoch, with deployment in online production systems.
The one-epoch overfitting problem has drawn widespread attention, especially in CTR and CVR estimation models in search, advertising, and recommendation domains. These models which rely heavily on large-scale sparse categorical features, often suffer a significant decline in performance when trained for multiple epochs. Although recent studies have proposed heuristic solutions, they have not clearly identified the fundamental cause of this phenomenon. In this work, we provide a theoretical analysis that explains why overfitting occurs in models that use large-scale sparse categorical features. Based on this analysis, we propose an adaptive regularization method to address it. Our approach not only prevents the severe performance degradation observed during multi-epoch training, but also improves model performance within a single epoch. This method has already been deployed in online production systems.