Long-form RewardBench: Evaluating Reward Models for Long-form Generation
This work addresses a critical gap for researchers and practitioners in AI alignment by providing the first benchmark for long-form reward modeling, though it is incremental as it builds on existing reward modeling benchmarks.
The authors tackled the lack of benchmarks for evaluating reward models in long-form generation by introducing Long-form RewardBench, a testbed covering five subtasks, and found that current models still lack capabilities in this area, with classifiers showing better generalizability than generative models.
The widespread adoption of reinforcement learning-based alignment highlights the growing importance of reward models. Various benchmarks have been built to evaluate reward models in various domains and scenarios. However, a significant gap remains in assessing reward models for long-form generation, despite its critical role in real-world applications. To bridge this, we introduce Long-form RewardBench, the first reward modeling testbed specifically designed for long-form generation. Our benchmark encompasses five key subtasks: QA, RAG, Chat, Writing, and Reasoning. We collected instruction and preference data through a meticulously designed multi-stage data collection process, and conducted extensive experiments on 20+ mainstream reward models, including both classifiers and generative models. Our findings reveal that current models still lack long-form reward modeling capabilities. Furthermore, we designed a novel Long-form Needle-in-a-Haystack Test, which revealed a correlation between reward modeling performance and the error's position within a response, as well as the overall response length, with distinct characteristics observed between classification and generative models. Finally, we demonstrate that classifiers exhibit better generalizability compared to generative models trained on the same data. As the first benchmark for long-form reward modeling, this work aims to offer a robust platform for visualizing progress in this crucial area.