Towards a Theoretical Understanding to the Generalization of RLHF
This provides theoretical evidence for the generalization of RLHF in aligning LLMs with human intent, addressing a gap in high-dimensional settings, but it is incremental as it builds on existing frameworks like algorithmic stability.
The paper tackles the lack of theoretical understanding of generalization in Reinforcement Learning from Human Feedback (RLHF) for Large Language Models, proving a generalization bound of order O(n^{-1/2}) under a feature coverage condition for policy models, which extends to gradient-based algorithms like Gradient Ascent and Stochastic Gradient Ascent.
Reinforcement Learning from Human Feedback (RLHF) and its variants have emerged as the dominant approaches for aligning Large Language Models with human intent. While empirically effective, the theoretical generalization properties of these methods in high-dimensional settings remain to be explored. To this end, we build the generalization theory on RLHF of LLMs under the linear reward model, through the framework of algorithmic stability. In contrast to the existing works built upon the consistency of maximum likelihood estimations on reward model, our analysis is presented under an end-to-end learning framework, which is consistent with practice. Concretely, we prove that under a key \textbf{feature coverage} condition, the empirical optima of policy model have a generalization bound of order $\mathcal{O}(n^{-\frac{1}{2}})$. Moreover, the results can be extrapolated to parameters obtained by gradient-based learning algorithms, i.e., Gradient Ascent (GA) and Stochastic Gradient Ascent (SGA). Thus, we argue that our results provide new theoretical evidence for the empirically observed generalization of LLMs after RLHF.