Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs
This work addresses the problem of enhancing ASR performance for Chinese speech recognition using LLMs, representing an incremental advancement in multimodal integration.
The paper tackled the challenge of integrating large language models (LLMs) with automatic speech recognition (ASR) by proposing a pre-training approach on Pinyin embedding sequences to generate Chinese characters, resulting in a 9.5% to 19.0% relative improvement in ASR tasks on the AISHELL-1 corpus.
The integration of large language models (LLMs) with pre-trained speech models has opened up new avenues in automatic speech recognition (ASR). While LLMs excel in multimodal understanding tasks, effectively leveraging their capabilities for ASR remains a significant challenge. This paper presents a novel training approach to enhance LLM performance in ASR tasks. We propose pre-training LLMs on Pinyin embedding sequences, which represent pronunciation features, to generate corresponding Chinese characters. This step enables the LLM to adapt to generating text from pronunciation features before encountering real speech data. Furthermore, we fine-tune the LoRA parameters to enhance the LLM's understanding of speech modality information. In AISHELL-1 corpus, our approach yields a 9.5% relative improvement in ASR tasks compared to the baseline without Pinyi-to-Character pre-training. Additionally, incorporating auxiliary text data for Pinyi-to-Character pre-training further boosts performance, achieving a 19.0% relative improvement.