Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives
This provides a standardized benchmark for CTR prediction in online advertising, facilitating model development and evaluation for practitioners, though it is incremental as it builds on existing datasets and models.
The researchers tackled the lack of standardized evaluation for click-through rate (CTR) prediction models by creating Bench-CTR, a unified benchmark platform with datasets, protocols, and guidelines, and found that high-order models generally outperform low-order ones, LLM-based models achieve comparable performance with only 2% of training data, and progress in CTR prediction has slowed since 2016.
This research designs a unified architecture of CTR prediction benchmark (Bench-CTR) platform that offers flexible interfaces with datasets and components of a wide range of CTR prediction models. Moreover, we construct a comprehensive system of evaluation protocols encompassing real-world and synthetic datasets, a taxonomy of metrics, standardized procedures and experimental guidelines for calibrating the performance of CTR prediction models. Furthermore, we implement the proposed benchmark platform and conduct a comparative study to evaluate a wide range of state-of-the-art models from traditional multivariate statistical to modern large language model (LLM)-based approaches on three public datasets and two synthetic datasets. Experimental results reveal that, (1) high-order models largely outperform low-order models, though such advantage varies in terms of metrics and on different datasets; (2) LLM-based models demonstrate a remarkable data efficiency, i.e., achieving the comparable performance to other models while using only 2% of the training data; (3) the performance of CTR prediction models has achieved significant improvements from 2015 to 2016, then reached a stage with slow progress, which is consistent across various datasets. This benchmark is expected to facilitate model development and evaluation and enhance practitioners' understanding of the underlying mechanisms of models in the area of CTR prediction. Code is available at https://github.com/NuriaNinja/Bench-CTR.