RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution
This addresses a bottleneck in RLHF for aligning LLMs with human preferences, offering an incremental improvement by enhancing token-level guidance without additional training costs.
The paper tackles the problem of sparse and delayed rewards in reinforcement learning from human feedback (RLHF) for large language models by proposing RED, a reward redistribution method that assigns token-level rewards using an off-the-shelf reward model, leading to more precise performance improvements as demonstrated across diverse datasets and tasks.
Reinforcement learning from human feedback (RLHF) offers a promising approach to aligning large language models (LLMs) with human preferences. Typically, a reward model is trained or supplied to act as a proxy for humans in evaluating generated responses during the reinforcement training phase. However, current reward models operate as sequence-to-one models, allocating a single, sparse, and delayed reward to an entire output sequence. This approach may overlook the significant contributions of individual tokens toward the desired outcome. To this end, we propose a more fine-grained, token-level guidance approach for RL training. Specifically, we introduce RED, a novel reward redistribition method that evaluates and assigns specific credit to each token using an off-the-shelf reward model. Utilizing these fine-grained rewards enhances the model's understanding of language nuances, leading to more precise performance improvements. Notably, our method does not require modifying the reward model or introducing additional training steps, thereby incurring minimal computational costs. Experimental results across diverse datasets and tasks demonstrate the superiority of our approach.