DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering
This addresses the challenge of building personal assistants for low-resource languages and scenarios with out-of-vocabulary words, though it is incremental as it builds on existing textless SQA approaches.
The paper tackles the problem of spoken question answering (SQA) without relying on ASR transcripts, which is difficult due to issues like recognition errors and data scarcity, by proposing DUAL, a method that uses unlabeled data for pre-training and fine-tuning to predict answer intervals directly from speech, achieving results comparable to ASR-based cascades and robustness on real-world data.
Spoken Question Answering (SQA) is to find the answer from a spoken document given a question, which is crucial for personal assistants when replying to the queries from the users. Existing SQA methods all rely on Automatic Speech Recognition (ASR) transcripts. Not only does ASR need to be trained with massive annotated data that are time and cost-prohibitive to collect for low-resourced languages, but more importantly, very often the answers to the questions include name entities or out-of-vocabulary words that cannot be recognized correctly. Also, ASR aims to minimize recognition errors equally over all words, including many function words irrelevant to the SQA task. Therefore, SQA without ASR transcripts (textless) is always highly desired, although known to be very difficult. This work proposes Discrete Spoken Unit Adaptive Learning (DUAL), leveraging unlabeled data for pre-training and fine-tuned by the SQA downstream task. The time intervals of spoken answers can be directly predicted from spoken documents. We also release a new SQA benchmark corpus, NMSQA, for data with more realistic scenarios. We empirically showed that DUAL yields results comparable to those obtained by cascading ASR and text QA model and robust to real-world data. Our code and model will be open-sourced.