Emphasis Sensitivity in Speech Representations
This addresses the problem of understanding prosodic emphasis encoding in speech models for researchers in speech processing, but it is incremental as it builds on prior work with a new analytical method.
The paper tackled the problem of whether modern speech models encode prosodic emphasis by proposing a residual-based framework to analyze differences between neutral and emphasized word representations, finding that residuals correlate with duration changes and show structured encoding, with ASR fine-tuned models having up to 50% more compact subspaces.
This work investigates whether modern speech models are sensitive to prosodic emphasis - whether they encode emphasized and neutral words in systematically different ways. Prior work typically relies on isolated acoustic correlates (e.g., pitch, duration) or label prediction, both of which miss the relational structure of emphasis. This paper proposes a residual-based framework, defining emphasis as the difference between paired neutral and emphasized word representations. Analysis on self-supervised speech models shows that these residuals correlate strongly with duration changes and perform poorly at word identity prediction, indicating a structured, relational encoding of prosodic emphasis. In ASR fine-tuned models, residuals occupy a subspace up to 50% more compact than in pre-trained models, further suggesting that emphasis is encoded as a consistent, low-dimensional transformation that becomes more structured with task-specific learning.