Learning Guarantee of Reward Modeling Using Deep Neural Networks
This work provides theoretical guarantees for reward modeling in reinforcement learning from human feedback, addressing a foundational problem for AI alignment researchers, though it appears incremental in extending existing learning theory to this specific context.
The paper tackles the problem of reward modeling with pairwise comparison data using deep neural networks by establishing a novel non-asymptotic regret bound that depends on network architecture. It introduces a margin-type condition for clear human beliefs, leading to a sharper regret bound that explains the empirical efficiency of Reinforcement Learning from Human Feedback.
In this work, we study the learning theory of reward modeling with pairwise comparison data using deep neural networks. We establish a novel non-asymptotic regret bound for deep reward estimators in a non-parametric setting, which depends explicitly on the network architecture. Furthermore, to underscore the critical importance of clear human beliefs, we introduce a margin-type condition that assumes the conditional winning probability of the optimal action in pairwise comparisons is significantly distanced from 1/2. This condition enables a sharper regret bound, which substantiates the empirical efficiency of Reinforcement Learning from Human Feedback and highlights clear human beliefs in its success. Notably, this improvement stems from high-quality pairwise comparison data implied by the margin-type condition, is independent of the specific estimators used, and thus applies to various learning algorithms and models.