AV-Dialog: Spoken Dialogue Models with Audio-Visual Input
This addresses the challenge of building robust spoken dialogue agents for real-world, noisy environments, representing a novel method for a known bottleneck.
The paper tackled the problem of dialogue models failing in noisy, multi-speaker environments by introducing AV-Dialog, a multimodal framework using audio and visual cues, which reduced transcription errors and improved turn-taking prediction and human-rated dialogue quality.
Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for {spoken} dialogue agents that perform {robustly} in real-world, noisy environments.