GRASS: Unified Generation Model for Speech-to-Semantic Tasks
This addresses the problem of handling diverse speech understanding tasks efficiently for AI and speech processing applications, though it is incremental as it builds on instruction fine-tuning techniques.
The paper tackles speech-to-semantic tasks by introducing a unified end-to-end framework that generates target text based on prompts for audio data, achieving state-of-the-art results on benchmarks like speech named entity recognition and sentiment analysis after fine-tuning, with competitive zero-shot and few-shot performance.
This paper explores the instruction fine-tuning technique for speech-to-semantic tasks by introducing a unified end-to-end (E2E) framework that generates target text conditioned on a task-related prompt for audio data. We pre-train the model using large and diverse data, where instruction-speech pairs are constructed via a text-to-speech (TTS) system. Extensive experiments demonstrate that our proposed model achieves state-of-the-art (SOTA) results on many benchmarks covering speech named entity recognition, speech sentiment analysis, speech question answering, and more, after fine-tuning. Furthermore, the proposed model achieves competitive performance in zero-shot and few-shot scenarios. To facilitate future work on instruction fine-tuning for speech-to-semantic tasks, we release our instruction dataset and code.