Speak or Stay Silent: Context-Aware Turn-Taking in Multi-Party Dialogue
This addresses a key usability issue for voice AI assistants in multi-party settings, though it is incremental as it builds on existing methods with new training data.
The paper tackles the problem of AI assistants being disruptive in multi-party dialogues by speaking on every pause, and proposes a supervised fine-tuning method that improves balanced accuracy by up to 23 percentage points for context-aware turn-taking.
Existing voice AI assistants treat every detected pause as an invitation to speak. This works in dyadic dialogue, but in multi-party settings, where an AI assistant participates alongside multiple speakers, pauses are abundant and ambiguous. An assistant that speaks on every pause becomes disruptive rather than useful. In this work, we formulate context-aware turn-taking: at every detected pause, given the full conversation context, our method decides whether the assistant should speak or stay silent. We introduce a benchmark of over 120K labeled conversations spanning three multi-party corpora. Evaluating eight recent large language models, we find that they consistently fail at context-aware turn-taking under zero-shot prompting. We then propose a supervised fine-tuning approach with reasoning traces, improving balanced accuracy by up to 23 percentage points. Our findings suggest that context-aware turn-taking is not an emergent capability; it must be explicitly trained.