Abacus: Self-Supervised Event Counting-Aligned Distributional Pretraining for Sequential User Modeling
This work addresses the problem of user modeling for display advertising systems, offering an incremental improvement by combining event-counting statistics with sequential learning.
The paper tackles the challenge of modeling sparse and stochastic user purchase behavior in display advertising by introducing Abacus, a self-supervised pretraining method that predicts event frequency distributions, resulting in up to +6.1% AUC improvement over baselines.
Modeling user purchase behavior is a critical challenge in display advertising systems, necessary for real-time bidding. The difficulty arises from the sparsity of positive user events and the stochasticity of user actions, leading to severe class imbalance and irregular event timing. Predictive systems usually rely on hand-crafted "counter" features, overlooking the fine-grained temporal evolution of user intent. Meanwhile, current sequential models extract direct sequential signal, missing useful event-counting statistics. We enhance deep sequential models with self-supervised pretraining strategies for display advertising. Especially, we introduce Abacus, a novel approach of predicting the empirical frequency distribution of user events. We further propose a hybrid objective unifying Abacus with sequential learning objectives, combining stability of aggregated statistics with the sequence modeling sensitivity. Experiments on two real-world datasets show that Abacus pretraining outperforms existing methods accelerating downstream task convergence, while hybrid approach yields up to +6.1% AUC compared to the baselines.