Unifying Acoustic Features and Text with Multimodal LLMs for Neurodegenerative Screening
This work addresses the challenge of neurodegenerative disease staging by unifying heterogeneous data types, offering a scalable and non-invasive screening approach.
NeurMLLM, a multimodal generative framework integrating acoustic features and text via a large language model, achieves strong performance in fine-grained staging of Alzheimer's and Parkinson's diseases, outperforming classical ML and existing LLM-based methods on the Bridge2AI-Voice dataset.
Voice-based screening offers a scalable and non-invasive way to assess neurodegenerative diseases such as Alzheimer's disease (AD) and Parkinson's disease (PD), but their staging remains challenging due to the difficulty of integrating heterogeneous data. This paper presents NeurMLLM, an efficient multimodal generative framework for neurodegenerative disease staging. NeurMLLM first encodes the spectrograms and Mel-frequency cepstral coefficients of audio data with vision transformers and projects their representations into the embedding space of a large language model (LLM), where they are concatenated with transcript and demographic instruction tokens as a single unified sequence. The LLM is then instruction-tuned via Low-Rank Adaptation using task prompts to autoregressively predict a constrained label token, enabling a generative classification. By evaluating on the Bridge2AI-Voice dataset for fine-grained staging of AD and PD, we observe that NeurMLLM achieves strong performance, consistently outperforming classical machine learning methods and existing LLM-based approaches. The results show the high potential of multimodal LLMs in neurodegenerative disease staging, improving staging accuracy and supporting accessible deployment.