Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
This work addresses the evaluation problem for reward models in AI alignment, offering a more detailed and interpretable method, though it is incremental as it builds on existing evaluation approaches.
The paper tackles the challenge of evaluating reward models by introducing a multi-dimensional probing method and benchmark (MRMBench), which strongly correlates with alignment performance in large language models and reveals that reward models often struggle with multiple preference dimensions.
Previous methods evaluate reward models by testing them on a fixed pairwise ranking test set, but they typically do not provide performance information on each preference dimension. In this work, we address the evaluation challenge of reward models by probing preference representations. To confirm the effectiveness of this evaluation method, we construct a Multi-dimensional Reward Model Benchmark (MRMBench), a collection of six probing tasks for different preference dimensions. We design it to favor and encourage reward models that better capture preferences across different dimensions. Furthermore, we introduce an analysis method, inference-time probing, which identifies the dimensions used during the reward prediction and enhances its interpretability. Through extensive experiments, we find that MRMBench strongly correlates with the alignment performance of large language models (LLMs), making it a reliable reference for developing advanced reward models. Our analysis of MRMBench evaluation results reveals that reward models often struggle to capture preferences across multiple dimensions, highlighting the potential of multi-objective optimization in reward modeling. Additionally, our findings show that the proposed inference-time probing method offers a reliable metric for assessing the confidence of reward predictions, which ultimately improves the alignment of LLMs.