Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understanding
This addresses the limitation of audio models in diagnostic healthcare by enabling clinically-aware tools, though it is incremental as it builds on existing alignment methods.
The paper tackles the problem of pre-trained audio models lacking clinical understanding in medical audio tasks by introducing AcuLa, a post-training framework that aligns audio encoders with a medical language model, resulting in state-of-the-art performance across 18 cardio-respiratory tasks, improving mean AUROC from 0.68 to 0.79 and boosting COVID-19 cough detection AUROC from 0.55 to 0.89.
Pre-trained audio models excel at detecting acoustic patterns in auscultation sounds but often fail to grasp their clinical significance, limiting their use and performance in diagnostic tasks. To bridge this gap, we introduce AcuLa (Audio-Clinical Understanding via Language Alignment), a lightweight post-training framework that instills semantic understanding into any audio encoder by aligning it with a medical language model, which acts as a "semantic teacher." To enable alignment at scale, we construct a large-scale dataset by leveraging off-the-shelf large language models to translate the rich, structured metadata accompanying existing audio recordings into coherent clinical reports. Our alignment strategy combines a representation-level contrastive objective with a self-supervised modeling, ensuring that the model learns clinical semantics while preserving fine-grained temporal cues. AcuLa achieves state-of-the-art results across 18 diverse cardio-respiratory tasks from 10 different datasets, improving the mean AUROC on classification benchmarks from 0.68 to 0.79 and, on the most challenging COVID-19 cough detection task, boosting the AUROC from 0.55 to 0.89. Our work demonstrates that this audio-language alignment transforms purely acoustic models into clinically-aware diagnostic tools, establishing a novel paradigm for enhancing physiological understanding in audio-based health monitoring.