Speech Recognition With LLMs Adapted to Disordered Speech Using Reinforcement Learning
This work addresses improving speech recognition for disordered speech, offering an incremental tuning strategy for LLMs in this domain.
The paper tackles adapting large language models (LLMs) to disordered speech by using reinforcement learning with custom rewards, resulting in substantially better performance than supervised fine-tuning when adapting to different speech settings, though it does not outperform existing speech recognition systems.
We introduce a large language model (LLM) capable of processing speech inputs and show that tuning it further with reinforcement learning on human preference (RLHF) enables it to adapt better to disordered speech than traditional fine-tuning. Our method replaces low-frequency text tokens in an LLM's vocabulary with audio tokens and enables the model to recognize speech by fine-tuning it on speech with transcripts. We then use RL with rewards based on syntactic and semantic accuracy measures generalizing the LLM further to recognize disordered speech. While the resulting LLM does not outperform existing systems for speech recognition, we find that tuning with reinforcement learning using custom rewards leads to substantially better performance than supervised fine-tuning of the language model, specifically when adapting to speech in a different setting. This presents a compelling alternative tuning strategy for speech recognition using large language models.