Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO
This work addresses the problem of spoken dialog systems in noisy, multi-speaker environments, which is relevant for real-world applications like smart assistants in social settings, but the contribution is incremental as it builds on existing LLM and RL techniques.
The paper introduces Cocktail-Talker, a speech LLM framework for multi-speaker dialog modeling in noisy social environments, using action tokens to decide whether to respond, listen, or ignore. It is trained with supervised finetuning and reinforcement learning, and includes a data pipeline for generating realistic multi-speaker dialogs. The result is a step toward more natural and selective spoken dialog systems in complex social settings.
Spoken dialog systems are typically designed for clean, dyadic interactions in which a single user and an assistant take turns speaking. Real-world social conversations, however, are often more ambiguous: multiple speakers may participate in the same conversation amid irrelevant speech and background noise. Each utterance may be directed to the assistant, addressed to another speaker, or completely irrelevant. In such settings, the assistant must decide not only what to say, but also whether to speak at all. In this paper, we introduce Cocktail-Talker, a speech LLM framework for multi-speaker spoken dialog modeling in noisy social environments. We model the assistant's behavior with three action tokens: <|respond|>, <|listen|>, and <|ignore|>, placed before a response or silence. Cocktail-Talker is trained via supervised finetuning and reinforcement learning to generate the appropriate action token and, only in <|respond|> mode, a speech response. To prepare the training data, we develop Cocktail-DialogGen, an LLM-based data pipeline that simulates realistic multi-speaker dialogs with speaker roles across diverse social settings. Together, these components take a step toward spoken dialog systems that interact more naturally and selectively in complex social environments.