Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities
This addresses the problem of limited evaluation methods for researchers and developers working on spoken dialogue models, though it is incremental as it builds on existing benchmarks by adding full-duplex-specific metrics.
The authors tackled the lack of systematic evaluation for full-duplex spoken dialogue models by introducing Full-Duplex-Bench, a benchmark that assesses interactive behaviors like pause handling and turn-taking, resulting in a framework with automatic metrics for reproducible assessment.
Spoken dialogue modeling poses challenges beyond text-based language modeling, requiring real-time interaction, turn-taking, and backchanneling. While most Spoken Dialogue Models (SDMs) operate in half-duplex mode-processing one turn at a time - emerging full-duplex SDMs can listen and speak simultaneously, enabling more natural conversations. However, current evaluations remain limited, focusing mainly on turn-based metrics or coarse corpus-level analyses. To address this, we introduce Full-Duplex-Bench, a benchmark that systematically evaluates key interactive behaviors: pause handling, backchanneling, turn-taking, and interruption management. Our framework uses automatic metrics for consistent, reproducible assessment and provides a fair, fast evaluation setup. By releasing our benchmark and code, we aim to advance spoken dialogue modeling and foster the development of more natural and engaging SDMs.