Speech foundation models in healthcare: Effect of layer selection on pathological speech feature prediction
This work addresses the need for accurate, automatic speech assessments to aid in diagnosing and treating neurological conditions, though it is incremental as it focuses on optimizing existing foundation models for a specific task.
The study tackled the problem of predicting pathological speech features for healthcare applications by investigating the effect of layer selection in speech foundation models, finding that optimal layer selection improves balanced accuracy by up to 15.8% compared to the worst layer, and a learned weighted sum method offers strong generalization with only a 1.5% performance drop on out-of-distribution data.
Accurately extracting clinical information from speech is critical to the diagnosis and treatment of many neurological conditions. As such, there is interest in leveraging AI for automatic, objective assessments of clinical speech to facilitate diagnosis and treatment of speech disorders. We explore transfer learning using foundation models, focusing on the impact of layer selection for the downstream task of predicting pathological speech features. We find that selecting an optimal layer can greatly improve performance (~15.8% increase in balanced accuracy per feature as compared to worst layer, ~13.6% increase as compared to final layer), though the best layer varies by predicted feature and does not always generalize well to unseen data. A learned weighted sum offers comparable performance to the average best layer in-distribution (only ~1.2% lower) and had strong generalization for out-of-distribution data (only 1.5% lower than the average best layer).