Permutation Equivariant Model-based Offline Reinforcement Learning for Auto-bidding
This addresses the challenge of improving auto-bidding performance in online advertising by expanding state coverage beyond fixed datasets, though it appears incremental as it builds on existing offline and model-based RL techniques.
The paper tackled the problem of limited state coverage in offline RL for auto-bidding by introducing a model-based RL approach that learns an environment model from real data to bridge the simulator-reality gap, resulting in outperforming state-of-the-art methods in real-world experiments.
Reinforcement learning (RL) for auto-bidding has shifted from using simplistic offline simulators (Simulation-based RL Bidding, SRLB) to offline RL on fixed real datasets (Offline RL Bidding, ORLB). However, ORLB policies are limited by the dataset's state space coverage, offering modest gains. While SRLB expands state coverage, its simulator-reality gap risks misleading policies. This paper introduces Model-based RL Bidding (MRLB), which learns an environment model from real data to bridge this gap. MRLB trains policies using both real and model-generated data, expanding state coverage beyond ORLB. To ensure model reliability, we propose: 1) A permutation equivariant model architecture for better generalization, and 2) A robust offline Q-learning method that pessimistically penalizes model errors. These form the Permutation Equivariant Model-based Offline RL (PE-MORL) algorithm. Real-world experiments show that PE-MORL outperforms state-of-the-art auto-bidding methods.