Contrastive Learning for Task-Independent SpeechLLM-Pretraining
This work addresses the problem of efficient adaptation of LLMs to speech tasks for researchers and practitioners, offering a scalable solution that reduces data and computational requirements, though it is incremental in building on existing contrastive learning techniques.
The paper tackles the challenge of adapting large language models to speech processing tasks efficiently by proposing a two-stage training approach with task-independent speech pretraining using contrastive learning, which outperforms traditional methods and achieves superior performance on speech translation and question answering with only 10% of task-specific data.
Large language models (LLMs) excel in natural language processing but adapting these LLMs to speech processing tasks efficiently is not straightforward. Direct task-specific fine-tuning is limited by overfitting risks, data requirements, and computational costs. To address these challenges, we propose a scalable, two-stage training approach: (1) A task-independent speech pretraining stage using contrastive learning to align text and speech representations over all layers, followed by (2) a task-specific fine-tuning stage requiring minimal data. This approach outperforms traditional ASR pretraining and enables the model to surpass models specialized on speech translation and question answering while being trained on only 10% of the task-specific data.