Bootstrapping LLMs via Preference-Based Policy Optimization
This addresses the challenge of efficiently bootstrapping LLMs for alignment, representing a novel method rather than an incremental improvement.
The paper tackles the problem of aligning large language models with human preferences without extensive manual annotations by proposing a preference-based policy optimization framework that formulates learning as a min-max game between a policy and a constrained reward model. The method achieves state-of-the-art performance on five benchmarks, with theoretical guarantees including high-probability regret bounds.
Bootstrapping large language models (LLMs) through preference-based policy optimization offers a promising direction for aligning model behavior with human preferences without relying on extensive manual annotations. In this work, we propose a novel preference-based policy optimization (PbPO) framework that formulates the learning process as a min-max game between the main policy and a reward model (RM). The RM is constrained within a confidence set derived from preference data to ensure reliable exploitation. Our iterative online algorithm actively collects preference data through guided exploration of the evolving policy, enabling continual self-improvement of both the policy and the RM. We provide theoretical guarantees for our method, establishing high-probability regret bounds for both settings with sequence-level RM and token-level RM, demonstrating its effectiveness in bootstrapping LLMs. Extensive experiments on five benchmarks show that our approach consistently outperforms existing state-of-the-art preference optimization techniques.