Aligning Spoken Dialogue Models from User Interactions
This work addresses the challenge of adapting preference learning to real-time speech interactions, which is incremental as it extends text-based methods to a more complex domain.
The authors tackled the problem of aligning spoken dialogue models with user preferences in real-time conversations, by creating a large-scale dataset of 150,000 preference pairs and finetuning a speech-to-speech model, resulting in improved factual accuracy, safety, and contextual alignment.
We propose a novel preference alignment framework for improving spoken dialogue models on real-time conversations from user interactions. Current preference learning methods primarily focus on text-based language models, and are not directly suited to the complexities of real-time speech interactions, with richer dynamics (e.g. interruption, interjection) and no explicit segmentation between speaker turns.We create a large-scale dataset of more than 150,000 preference pairs from raw multi-turn speech conversations, annotated with AI feedback, to cover preferences over both linguistic content and temporal context variations. We leverage offline alignment methods to finetune a full-duplex autoregressive speech-to-speech model. Extensive experiments demonstrate that feedback on generic conversations can be consistently effective in improving spoken dialogue models to produce more factual, safer and more contextually aligned interactions. We deploy the finetuned model and conduct holistic human evaluations to assess the impact beyond single-turn conversations. Our findings shed light on the importance of a well-calibrated balance among various dynamics, crucial for natural real-time speech dialogue systems.