SpeechMedAssist: Efficiently and Effectively Adapting Speech Language Models for Medical Consultation
This addresses the problem of inefficient and data-scarce speech-based medical consultation systems for patients and healthcare providers, though it is incremental in adapting existing speech language models.
The paper tackles the challenge of adapting speech language models for medical consultations by proposing SpeechMedAssist, which uses a two-stage training paradigm to reduce the need for medical speech data to only 10k synthesized samples, resulting in a model that outperforms baselines in effectiveness and robustness.
Medical consultations are intrinsically speech-centric. However, most prior works focus on long-text-based interactions, which are cumbersome and patient-unfriendly. Recent advances in speech language models (SpeechLMs) have enabled more natural speech-based interaction, yet the scarcity of medical speech data and the inefficiency of directly fine-tuning on speech data jointly hinder the adoption of SpeechLMs in medical consultation. In this paper, we propose SpeechMedAssist, a SpeechLM natively capable of conducting speech-based multi-turn interactions with patients. By exploiting the architectural properties of SpeechLMs, we decouple the conventional one-stage training into a two-stage paradigm consisting of (1) Knowledge & Capability Injection via Text and (2) Modality Re-alignment with Limited Speech Data, thereby reducing the requirement for medical speech data to only 10k synthesized samples. To evaluate SpeechLMs for medical consultation scenarios, we design a benchmark comprising both single-turn question answering and multi-turn simulated interactions. Experimental results show that our model outperforms all baselines in both effectiveness and robustness in most evaluation settings.