Generalist Reward Models: Found Inside Large Language Models
This work addresses the costly alignment problem for LLM developers by providing a principled, efficient alternative to reward modeling, potentially impacting all of ML/AI.
The paper tackles the high cost of training reward models for aligning Large Language Models by discovering that a generalist reward model is already present within any LLM trained via next-token prediction, and proves it is theoretically equivalent to offline inverse reinforcement learning, enabling direct elicitation of a reward signal without further training. Experiments show the method outperforms existing LLM-as-a-judge approaches and can surpass explicitly trained reward models.
The alignment of Large Language Models (LLMs) is critically dependent on reward models trained on costly human preference data. While recent work explores bypassing this cost with AI feedback, these methods often lack a rigorous theoretical foundation. In this paper, we discover that a powerful generalist reward model is already latently present within any LLM trained via standard next-token prediction. We prove that this endogenous reward is not a heuristic, but is theoretically equivalent to a reward function learned through offline inverse reinforcement learning. This connection allows us to directly elicit a high-quality reward signal from a base (pre-trained or supervised fine-tuned) model without any further training. Critically, we also prove that subsequent reinforcement learning using this endogenous reward leads to a policy with a provably superior error bound compared to the base model. To our best knowledge, this is the first theoretical proof of the effectiveness of reinforcement learning for LLMs. Our experiments validate this theory, demonstrating that our method not only outperforms existing LLM-as-a-judge approaches but can also surpass explicitly trained reward models. These findings suggest that the reward modeling stage can be replaced by a principled method of eliciting the knowledge already captured during pre-training, heralding a more efficient, powerful, and scalable paradigm for LLMs alignment as well as multi-modal models.