Endpoint Anticipation for Low-Latency Spoken Dialogue
This work addresses the latency bottleneck in cascaded spoken dialogue architectures for real-time speech-to-speech interaction, enabling more natural conversational AI.
Endpoint Anticipation shifts from reactive turn-completion detection to proactive forecasting of end-of-turn signals, achieving up to 2.56 seconds advance prediction and demonstrating a 505 ms average latency reduction with 28.4% increased speculative computation in spoken dialogue systems.
While low-latency interaction is critical for spoken dialogue, cascaded architectures are often bottlenecked by reactive turn-completion detection. We propose Endpoint Anticipation, shifting from reactive detection to proactive forecasting of end-of-turn signals. Our speech-based model anticipates endpoints upto 2.56 seconds in advance, enabling speculative execution of LLM and TTS pipelines on partial context. We introduce metrics to quantify the trade-off between realized latency reduction and computational redundancy. Evaluation across conversational and task-oriented datasets shows our model consistently outperforms competitive VAP-based baselines. Integration with the Unmute framework demonstrates a 505 ms average latency reduction with a 28.4% increase in speculative computation, effectively masking sequential bottlenecks to enable complex reasoning in real-time speech-to-speech interaction.