Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modeling
This work addresses the need for efficient multimodal speech modeling in conversational assistants, though it appears incremental as it builds on existing methods with data-centric improvements.
The paper tackles the problem of enhancing multimodal understanding in conversational speech modeling by introducing a data-centric customization approach with a novel multi-task learning paradigm, achieving state-of-the-art performance on the Spoken-SQuAD benchmark using only 10% of training data.
Conversational assistants are increasingly popular across diverse real-world applications, highlighting the need for advanced multimodal speech modeling. Speech, as a natural mode of communication, encodes rich user-specific characteristics such as speaking rate and pitch, making it critical for effective interaction. Our work introduces a data-centric customization approach for efficiently enhancing multimodal understanding in conversational speech modeling. Central to our contributions is a novel multi-task learning paradigm that involves designing auxiliary tasks to utilize a small amount of speech data. Our approach achieves state-of-the-art performance on the Spoken-SQuAD benchmark, using only 10% of the training data with open-weight models, establishing a robust and efficient framework for audio-centric conversational modeling. We also introduce ASK-QA, the first dataset for multi-turn spoken dialogue with ambiguous user requests and dynamic evaluation inputs. Code and data forthcoming.