Do What I Say: A Spoken Prompt Dataset for Instruction-Following
This addresses the problem of unrealistic evaluation for SLLMs, which typically rely on text prompts, by providing a dataset for more accurate assessment, though it is incremental as it focuses on benchmarking rather than model development.
The authors tackled the lack of realistic speech-based evaluation for Speech Large Language Models (SLLMs) by introducing the DoWhatISay dataset, which includes human-recorded spoken and written prompts across 9 tasks and 11 languages, and found that text prompts consistently outperform spoken prompts except in speech output tasks.
Speech Large Language Models (SLLMs) have rapidly expanded, supporting a wide range of tasks. These models are typically evaluated using text prompts, which may not reflect real-world scenarios where users interact with speech. To address this gap, we introduce DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to pair with any existing benchmark for realistic evaluation of SLLMs under spoken instruction conditions. Spanning 9 tasks and 11 languages, it provides 10 prompt variants per task-language pair, across five styles. Using DOWIS, we benchmark state-of-the-art SLLMs, analyzing the interplay between prompt modality, style, language, and task type. Results show that text prompts consistently outperform spoken prompts, particularly for low-resource and cross-lingual settings. Only for tasks with speech output, spoken prompts do close the gap, highlighting the need for speech-based prompting in SLLM evaluation.