SLM: Bridge the thin gap between speech and text foundation models
This work addresses the problem of integrating multimodal AI capabilities for researchers and practitioners, offering an efficient adaptation method that is incremental in leveraging existing models.
The authors tackled the challenge of bridging speech and text foundation models by introducing a joint Speech and Language Model (SLM) that uses a simple adapter with only 1% of parameters, achieving strong performance on tasks like speech recognition and translation while enabling zero-shot instruction-following for diverse generation tasks.
We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation models to maximally preserves their capabilities, and only trains a simple adapter with just 1\% (156M) of the foundation models' parameters. This adaptation not only leads SLM to achieve strong performance on conventional tasks such as speech recognition (ASR) and speech translation (AST), but also introduces the novel capability of zero-shot instruction-following for more diverse tasks: given a speech input and a text instruction, SLM is able to perform unseen generation tasks including contextual biasing ASR using real-time context, dialog generation, speech continuation, and question answering, etc. Our approach demonstrates that the representational gap between pretrained speech and language models might be narrower than one would expect, and can be bridged by a simple adaptation mechanism. As a result, SLM is not only efficient to train, but also inherits strong capabilities already acquired in foundation models of different modalities.