MUSIC: MUlti-Step Instruction Contrast for Multi-Turn Reward Models
This addresses the problem of scalable evaluation for multi-turn interactions in LLMs, offering a more nuanced alternative to costly human evaluation, though it is incremental as it builds on existing reward model techniques.
The paper tackles the challenge of evaluating multi-turn conversations for LLMs by proposing MUSIC, an unsupervised data augmentation strategy that synthesizes contrastive conversation pairs across multiple turns, and shows that training a reward model with MUSIC outperforms baselines in alignment with LLM judges on multi-turn tasks without degrading single-turn performance.
Evaluating the quality of multi-turn conversations is crucial for developing capable Large Language Models (LLMs), yet remains a significant challenge, often requiring costly human evaluation. Multi-turn reward models (RMs) offer a scalable alternative and can provide valuable signals for guiding LLM training. While recent work has advanced multi-turn \textit{training} techniques, effective automated \textit{evaluation} specifically for multi-turn interactions lags behind. We observe that standard preference datasets, typically contrasting responses based only on the final conversational turn, provide insufficient signal to capture the nuances of multi-turn interactions. Instead, we find that incorporating contrasts spanning \textit{multiple} turns is critical for building robust multi-turn RMs. Motivated by this finding, we propose \textbf{MU}lti-\textbf{S}tep \textbf{I}nstruction \textbf{C}ontrast (MUSIC), an unsupervised data augmentation strategy that synthesizes contrastive conversation pairs exhibiting differences across multiple turns. Leveraging MUSIC on the Skywork preference dataset, we train a multi-turn RM based on the Gemma-2-9B-Instruct model. Empirical results demonstrate that our MUSIC-augmented RM outperforms baseline methods, achieving higher alignment with judgments from advanced proprietary LLM judges on multi-turn conversations, crucially, without compromising performance on standard single-turn RM benchmarks.