DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization
This work addresses the challenge of selecting high-quality training data for preference optimization in large language models, which is incremental as it builds on existing dataset analysis methods.
The authors tackled the problem of improving preference optimization (PO) performance by measuring the quality of response pairs in training datasets, using a new metric called DCRM that combines distance and reward margin to encourage minimal noisy and maximal desired differences. They found a correlation between higher DCRM and better learning outcomes, and their best-of-N^2 pairing method produced datasets that improved model performance on benchmarks like AlpacaEval, MT-Bench, and Arena-Hard over existing sets.
Recent research has attempted to associate preference optimization (PO) performance with the underlying preference datasets. In this work, our observation is that the differences between the preferred response $y^+$ and dispreferred response $y^-$ influence what LLMs can learn, which may not match the desirable differences to learn. Therefore, we use distance and reward margin to quantify these differences, and combine them to get Distance Calibrated Reward Margin (DCRM), a metric that measures the quality of a response pair for PO. Intuitively, DCRM encourages minimal noisy differences and maximal desired differences. With this, we study 3 types of commonly used preference datasets, classified along two axes: the source of the responses and the preference labeling function. We establish a general correlation between higher DCRM of the training set and better learning outcome. Inspired by this, we propose a best-of-$N^2$ pairing method that selects response pairs with the highest DCRM. Empirically, in various settings, our method produces training datasets that can further improve models' performance on AlpacaEval, MT-Bench, and Arena-Hard over the existing training sets.