Confidence Ranking for CTR Prediction
This work addresses the need for better model adaptation in online systems like advertisement platforms, though it appears incremental as it builds on existing ranking and optimization methods.
The paper tackles the problem of adapting models in large-scale real-world applications like ads and recommendation systems by proposing a Confidence Ranking framework that optimizes ranking functions with two models, resulting in outperforming all baselines on CTR prediction tasks across public and industrial datasets.
Model evolution and constant availability of data are two common phenomena in large-scale real-world machine learning applications, e.g. ads and recommendation systems. To adapt, the real-world system typically retrain with all available data and online learn with recently available data to update the models periodically with the goal of better serving performance. In this paper, we propose a novel framework, named Confidence Ranking, which designs the optimization objective as a ranking function with two different models. Our confidence ranking loss allows direct optimization of the logits output for different convex surrogate functions of metrics, e.g. AUC and Accuracy depending on the target task and dataset. Armed with our proposed methods, our experiments show that the introduction of confidence ranking loss can outperform all baselines on the CTR prediction tasks of public and industrial datasets. This framework has been deployed in the advertisement system of JD.com to serve the main traffic in the fine-rank stage.