General Exploratory Bonus for Optimistic Exploration in RLHF
This addresses the sample efficiency problem in RLHF for AI alignment researchers and practitioners, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackled the problem that existing exploratory bonus methods in reinforcement learning with human feedback unintentionally bias exploration toward conservative behavior rather than promoting discovery of uncertain regions. The result was the introduction of the General Exploratory Bonus framework, which provably satisfies the optimism principle and empirically outperforms baselines on alignment tasks across multiple divergence settings and large language model backbones.
Optimistic exploration is central to improving sample efficiency in reinforcement learning with human feedback, yet existing exploratory bonus methods to incentivize exploration often fail to realize optimism. We provide a theoretical analysis showing that current formulations, under KL or $α$-divergence regularization, unintentionally bias exploration toward high-probability regions of the reference model, thereby reinforcing conservative behavior instead of promoting discovery of uncertain regions. To address this pitfall, we introduce the General Exploratory Bonus (GEB), a novel theoretical framework that provably satisfies the optimism principle. GEB counteracts divergence-induced bias via reference-dependent reward regulation and unifies prior heuristic bonuses as special cases, while extending naturally across the full $α$-divergence family. Empirically, GEB consistently outperforms baselines on alignment tasks across multiple divergence settings and large language model backbones. These results demonstrate that GEB offers both a principled and practical solution for optimistic exploration in RLHF.