Optimal Bidding Strategy without Exploration in Real-time Bidding
This work addresses a practical challenge for advertisers in online advertising by providing a more robust bidding strategy that generalizes to new budget conditions, though it is incremental in improving existing methods.
The paper tackles the problem of evaluating and generalizing optimal bidding strategies in real-time bidding systems, where previous methods fail due to censored data and lack of adaptability to unseen budget constraints, and demonstrates significantly improved performance on real-world datasets.
Maximizing utility with a budget constraint is the primary goal for advertisers in real-time bidding (RTB) systems. The policy maximizing the utility is referred to as the optimal bidding strategy. Earlier works on optimal bidding strategy apply model-based batch reinforcement learning methods which can not generalize to unknown budget and time constraint. Further, the advertiser observes a censored market price which makes direct evaluation infeasible on batch test datasets. Previous works ignore the losing auctions to alleviate the difficulty with censored states; thus significantly modifying the test distribution. We address the challenge of lacking a clear evaluation procedure as well as the error propagated through batch reinforcement learning methods in RTB systems. We exploit two conditional independence structures in the sequential bidding process that allow us to propose a novel practical framework using the maximum entropy principle to imitate the behavior of the true distribution observed in real-time traffic. Moreover, the framework allows us to train a model that can generalize to the unseen budget conditions than limit only to those observed in history. We compare our methods on two real-world RTB datasets with several baselines and demonstrate significantly improved performance under various budget settings.