VocalBench-DF: A Benchmark for Evaluating Speech LLM Robustness to Disfluency
This addresses the problem of making Speech-LLMs inclusive for users with speech impairments, but it is incremental as it focuses on benchmarking and identifying bottlenecks rather than proposing new methods.
The paper tackles the problem of Speech-LLMs lacking robustness to speech disfluencies, particularly from conditions like Parkinson's disease, by introducing VocalBench-DF, a benchmark for evaluation; evaluation of 22 models shows substantial performance degradation, indicating limited real-world readiness.
While Speech Large Language Models (Speech-LLMs) show strong performance in many applications, their robustness is critically under-tested, especially to speech disfluency. Existing evaluations often rely on idealized inputs, overlooking common disfluencies, particularly those associated with conditions like Parkinson's disease. This work investigates whether current Speech-LLMs can maintain performance when interacting with users who have speech impairments. To facilitate this inquiry, we introduce VocalBench-DF, a framework for the systematic evaluation of disfluency across a multi-dimensional taxonomy. Our evaluation of 22 mainstream Speech-LLMs reveals substantial performance degradation, indicating that their real-world readiness is limited. Further analysis identifies phoneme-level processing and long-context modeling as primary bottlenecks responsible for these failures. Strengthening recognition and reasoning capability from components and pipelines can substantially improve robustness. These findings highlight the urgent need for new methods to improve disfluency handling and build truly inclusive Speech-LLMs