LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models
This addresses the problem of multimodal functionality deficits in LLMs for researchers and developers in speech and language processing, though it is incremental as it builds on existing datasets and methods.
The paper tackles the challenge of Spoken Question Answering (SQA) with Large Language Models (LLMs) by introducing the LibriSQA dataset with 107k SQA pairs and a lightweight framework, achieving significant results and demonstrating LLMs' capability in multimodal alignment.
While Large Language Models (LLMs) have demonstrated commendable performance across a myriad of domains and tasks, existing LLMs still exhibit a palpable deficit in handling multimodal functionalities, especially for the Spoken Question Answering (SQA) task which necessitates precise alignment and deep interaction between speech and text features. To address the SQA challenge on LLMs, we initially curated the free-form and open-ended LibriSQA dataset from Librispeech, comprising Part I with natural conversational formats and Part II encompassing multiple-choice questions followed by answers and analytical segments. Both parts collectively include 107k SQA pairs that cover various topics. Given the evident paucity of existing speech-text LLMs, we propose a lightweight, end-to-end framework to execute the SQA task on the LibriSQA, witnessing significant results. By reforming ASR into the SQA format, we further substantiate our framework's capability in handling ASR tasks. Our empirical findings bolster the LLMs' aptitude for aligning and comprehending multimodal information, paving the way for the development of universal multimodal LLMs. The dataset and demo can be found at https://github.com/ZihanZhaoSJTU/LibriSQA.