The Sound of Syntax: Finetuning and Comprehensive Evaluation of Language Models for Speech Pathology
This addresses the need for technological support to improve speech-language pathologists' productivity in clinical settings, though it's incremental as it focuses on benchmarking and fine-tuning existing models rather than developing new ones.
The paper tackles the problem of evaluating multimodal language models for speech pathology applications, where there's a shortage of clinicians relative to affected children, by creating the first comprehensive benchmark across five core use cases with 1,000 annotated data points each. The evaluation of 15 models revealed no single model consistently outperforms others, with systematic disparities like better performance on male speakers, but fine-tuning achieved over 10% improvements compared to base models.
According to the U.S. National Institutes of Health, more than 3.4 million children experience speech disorders that require clinical intervention. The number of speech-language pathologists (SLPs) is roughly 20 times fewer than the number of affected children, highlighting a significant gap in children's care and a pressing need for technological support that improves the productivity of SLPs. State-of-the-art multimodal language models (MLMs) show promise for supporting SLPs, but their use remains underexplored largely due to a limited understanding of their performance in high-stakes clinical settings. To address this gap, we collaborate with domain experts to develop a taxonomy of real-world use cases of MLMs in speech-language pathologies. Building on this taxonomy, we introduce the first comprehensive benchmark for evaluating MLM across five core use cases, each containing 1,000 manually annotated data points. This benchmark includes robustness and sensitivity tests under various settings, including background noise, speaker gender, and accent. Our evaluation of 15 state-of-the-art MLMs reveals that no single model consistently outperforms others across all tasks. Notably, we find systematic disparities, with models performing better on male speakers, and observe that chain-of-thought prompting can degrade performance on classification tasks with large label spaces and narrow decision boundaries. Furthermore, we study fine-tuning MLMs on domain-specific data, achieving improvements of over 10\% compared to base models. These findings highlight both the potential and limitations of current MLMs for speech-language pathology applications, underscoring the need for further research and targeted development.