Evaluating Reward Model Generalization via Pairwise Maximum Discrepancy Competitions
This addresses the challenge of assessing reward model generalization for AI alignment practitioners, offering a more faithful evaluation method, though it is incremental as it builds on existing RM evaluation approaches.
The paper tackled the problem of evaluating reward model generalization in open-world settings by introducing Pairwise Maximum Discrepancy Competition (PMDC), a dynamic framework that actively selects contentious test cases from an unlabeled prompt pool, resulting in substantial rank reshuffling for 10 representative RMs compared to conventional benchmarks.
Reward models (RMs) are central to aligning large language models, yet their practical effectiveness hinges on generalization to unseen prompts and shifting distributions. Most existing RM evaluations rely on static, pre-annotated preference datasets, which provide limited coverage and often fail to faithfully assess generalization in open-world settings. We introduce Pairwise Maximum Discrepancy Competition (PMDC), a dynamic and annotation-efficient framework for evaluating RM generalization using a large, unlabeled, open-domain prompt pool. PMDC actively selects prompt--response pairs that maximize disagreement between two RMs, yielding a compact set of highly contentious test cases. These cases are adjudicated by an oracle, and the resulting outcomes are aggregated via a Bradley--Terry model to produce a global ranking and pairwise win-rate landscape of RMs. We apply PMDC to re-evaluate 10 representative RMs and observe substantial rank reshuffling compared with conventional benchmarks. Qualitative analyses further uncover systematic generalization failures, providing valuable insights for improving reward modeling.