Instruction-Following Speech Recognition
This addresses the need for more adaptable and privacy-aware speech recognition systems for users, though it is incremental as it builds on existing ASR methods with a novel training approach.
The paper tackles the problem of conventional ASR models lacking flexibility for nuanced user interactions by introducing instruction-following speech recognition, training a Listen-Attend-Spell model to execute free-form text instructions, enabling tasks like transcript manipulation and summarization without predefined commands, and achieving this from scratch on Librispeech without LLMs or pre-trained modules.
Conventional end-to-end Automatic Speech Recognition (ASR) models primarily focus on exact transcription tasks, lacking flexibility for nuanced user interactions. With the advent of Large Language Models (LLMs) in speech processing, more organic, text-prompt-based interactions have become possible. However, the mechanisms behind these models' speech understanding and "reasoning" capabilities remain underexplored. To study this question from the data perspective, we introduce instruction-following speech recognition, training a Listen-Attend-Spell model to understand and execute a diverse set of free-form text instructions. This enables a multitude of speech recognition tasks -- ranging from transcript manipulation to summarization -- without relying on predefined command sets. Remarkably, our model, trained from scratch on Librispeech, interprets and executes simple instructions without requiring LLMs or pre-trained speech modules. It also offers selective transcription options based on instructions like "transcribe first half and then turn off listening," providing an additional layer of privacy and safety compared to existing LLMs. Our findings highlight the significant potential of instruction-following training to advance speech foundation models.