BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5
This addresses the problem of enabling real-time, multitask speech processing for AI applications, though it is incremental by building on existing SpeechLLM architectures.
The paper tackles the challenge of integrating speech understanding into large language models by proposing BESTOW, an architecture that combines GPT and T5 styles to achieve efficient, multitask capabilities and the first open-source streaming solution, demonstrating strong performance on tasks like ASR and AST with lower training costs.
Incorporating speech understanding capabilities into pretrained large-language models has become a vital research direction (SpeechLLM). The previous architectures can be categorized as: i) GPT-style, prepend speech prompts to the text prompts as a sequence of LLM inputs like a decoder-only model; ii) T5-style, introduce speech cross-attention to each layer of the pretrained LLMs. We propose BESTOW architecture to bring the BESt features from TwO Worlds into a single model that is highly efficient and has strong multitask capabilities. Moreover, there is no clear streaming solution for either style, especially considering the solution should generalize to speech multitask. We reformulate streamable SpeechLLM as a read-write policy problem and unifies the offline and streaming research with BESTOW architecture. Hence we demonstrate the first open-source SpeechLLM solution that enables Streaming and Multitask at scale (beyond ASR) at the same time. This streamable solution achieves very strong performance on a wide range of speech tasks (ASR, AST, SQA, unseen DynamicSuperb). It is end-to-end optimizable, with lower training/inference cost, and demonstrates LLM knowledge transferability to speech.