WavReward: Spoken Dialogue Models With Generalist Reward Evaluators
This addresses the evaluation gap for spoken dialogue models like GPT-4o-audio, which is important for researchers and developers in speech AI, though it is incremental as it builds on existing audio language models and reinforcement learning.
The paper tackles the problem of evaluating spoken dialogue models' conversational performance, which is difficult with text-based methods, by proposing WavReward, a reward feedback model based on audio language models that assesses both IQ and EQ. The result shows WavReward outperforms previous state-of-the-art models, improving objective accuracy from 53.4% to 91.5% and achieving an 83% margin in subjective A/B testing.
End-to-end spoken dialogue models such as GPT-4o-audio have recently garnered significant attention in the speech domain. However, the evaluation of spoken dialogue models' conversational performance has largely been overlooked. This is primarily due to the intelligent chatbots convey a wealth of non-textual information which cannot be easily measured using text-based language models like ChatGPT. To address this gap, we propose WavReward, a reward feedback model based on audio language models that can evaluate both the IQ and EQ of spoken dialogue systems with speech input. Specifically, 1) based on audio language models, WavReward incorporates the deep reasoning process and the nonlinear reward mechanism for post-training. By utilizing multi-sample feedback via the reinforcement learning algorithm, we construct a specialized evaluator tailored to spoken dialogue models. 2) We introduce ChatReward-30K, a preference dataset used to train WavReward. ChatReward-30K includes both comprehension and generation aspects of spoken dialogue models. These scenarios span various tasks, such as text-based chats, nine acoustic attributes of instruction chats, and implicit chats. WavReward outperforms previous state-of-the-art evaluation models across multiple spoken dialogue scenarios, achieving a substantial improvement about Qwen2.5-Omni in objective accuracy from 53.4$\%$ to 91.5$\%$. In subjective A/B testing, WavReward also leads by a margin of 83$\%$. Comprehensive ablation studies confirm the necessity of each component of WavReward. All data and code will be publicly at https://github.com/jishengpeng/WavReward after the paper is accepted.