Helen: Optimizing CTR Prediction Models with Frequency-wise Hessian Eigenvalue Regularization
This work addresses optimization challenges in CTR prediction for online advertising and recommendation systems, offering a novel optimizer that improves model performance, though it is incremental as it builds on sharpness-aware minimization techniques.
The paper tackles the limited performance improvements in Click-Through Rate (CTR) prediction models by identifying a correlation between feature frequency and sharp local minima, and proposes Helen, an optimizer that uses frequency-wise Hessian eigenvalue regularization to address this, showing clear advantages over existing methods on benchmark datasets.
Click-Through Rate (CTR) prediction holds paramount significance in online advertising and recommendation scenarios. Despite the proliferation of recent CTR prediction models, the improvements in performance have remained limited, as evidenced by open-source benchmark assessments. Current researchers tend to focus on developing new models for various datasets and settings, often neglecting a crucial question: What is the key challenge that truly makes CTR prediction so demanding? In this paper, we approach the problem of CTR prediction from an optimization perspective. We explore the typical data characteristics and optimization statistics of CTR prediction, revealing a strong positive correlation between the top hessian eigenvalue and feature frequency. This correlation implies that frequently occurring features tend to converge towards sharp local minima, ultimately leading to suboptimal performance. Motivated by the recent advancements in sharpness-aware minimization (SAM), which considers the geometric aspects of the loss landscape during optimization, we present a dedicated optimizer crafted for CTR prediction, named Helen. Helen incorporates frequency-wise Hessian eigenvalue regularization, achieved through adaptive perturbations based on normalized feature frequencies. Empirical results under the open-source benchmark framework underscore Helen's effectiveness. It successfully constrains the top eigenvalue of the Hessian matrix and demonstrates a clear advantage over widely used optimization algorithms when applied to seven popular models across three public benchmark datasets on BARS. Our code locates at github.com/NUS-HPC-AI-Lab/Helen.