SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation
This work addresses the challenge of adapting large language models to handle speech tasks, which is incremental as it builds on existing methods with novel adaptations.
The authors tackled the problem of integrating speech inputs into language models for automatic speech recognition and translation, achieving performance on par with task-specific baselines and demonstrating zero-shot in-context learning capabilities.
We present a novel Speech Augmented Language Model (SALM) with {\em multitask} and {\em in-context} learning capabilities. SALM comprises a frozen text LLM, a audio encoder, a modality adapter module, and LoRA layers to accommodate speech input and associated task instructions. The unified SALM not only achieves performance on par with task-specific Conformer baselines for Automatic Speech Recognition (ASR) and Speech Translation (AST), but also exhibits zero-shot in-context learning capabilities, demonstrated through keyword-boosting task for ASR and AST. Moreover, {\em speech supervised in-context training} is proposed to bridge the gap between LLM training and downstream speech tasks, which further boosts the in-context learning ability of speech-to-text models. Proposed model is open-sourced via NeMo toolkit.