StressTest: Can YOUR Speech LM Handle the Stress?
This addresses a gap in evaluating and developing speech-aware language models for understanding nuanced meaning through sentence stress, which is important for applications like spoken question answering, but it is incremental as it builds on existing SLMs with a new benchmark and training data.
The authors tackled the problem that speech-aware language models (SLMs) often overlook sentence stress, which is crucial for understanding meaning and intent in speech, by introducing StressTest, a benchmark for evaluating stress-based meaning distinction, and found that leading SLMs perform poorly on such tasks. They proposed a data generation pipeline to create Stress-17k, a training set, and their finetuned model, StresSLM, notably outperformed existing SLMs on sentence stress reasoning and detection, generalizing well to real recordings.
Sentence stress refers to emphasis on words within a spoken utterance to highlight or contrast an idea. It is often used to imply an underlying intention not explicitly stated. Recent speech-aware language models (SLMs) have enabled direct audio processing, allowing models to access the full richness of speech to perform audio reasoning tasks such as spoken question answering. Despite the crucial role of sentence stress in shaping meaning and intent, it remains largely overlooked in evaluation and development of SLMs. We address this gap by introducing StressTest, a benchmark designed to evaluate models' ability to distinguish between meanings of speech based on the stress pattern. We evaluate leading SLMs, and find that despite their overall capabilities, they perform poorly on such tasks. Hence, we propose a novel data generation pipeline, and create Stress-17k, a training set that simulates change of meaning implied by stress variation. Results suggest, that our finetuned model, StresSLM, generalizes well to real recordings and notably outperforms existing SLMs on sentence stress reasoning and detection. Models, code, data, samples - pages.cs.huji.ac.il/adiyoss-lab/stresstest.