The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASR$\rightarrow$LLM Pipelines?
This reveals that many deployed speech LLMs may be inefficient cascades rather than novel models, with performance degrading under noise (up to 7.6% at 0 dB), which is incremental for speech AI efficiency.
The paper investigates whether current speech LLMs are behaviorally equivalent to simple ASR→LLM cascades, finding that most (like Ultravox) are statistically indistinguishable from matched cascades (κ=0.93) and collapse in accuracy when text representations are erased, while Qwen2-Audio shows divergence, indicating architecture-dependence.
Current speech LLMs largely perform implicit ASR: on tasks solvable from a transcript, they are behaviorally and mechanistically equivalent to simple Whisper$\to$LLM cascades. We show this through matched-backbone testing across four speech LLMs and six tasks, controlling for the LLM backbone for the first time. Ultravox is statistically indistinguishable from its matched cascade ($κ{=}0.93$); logit lens reveals literal text emerging in hidden states; LEACE concept erasure confirms text representations are causally necessary in both architectures tested, collapsing accuracy to near-zero. Qwen2-Audio genuinely diverges, revealing cascade equivalence is architecture-dependent, not universal. For most deployed use cases, current speech LLMs are expensive cascades, and under noise, they are worse ones, with clean-condition advantages reversing by up to 7.6% at 0 dB.