Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation
For researchers building spoken dialogue systems from text LLM backbones, this work provides design principles for speech token representations to improve reasoning, though the gains are incremental.
The paper identifies temporal-granularity mismatch between speech and text tokens as a cause of reasoning degradation in spoken dialogue models, and proposes factorized FSQ and a lightweight audio LM head to enable low frame rates. Sweeping frame rates from 50 to 2.08 Hz, they find optimal speech QA performance at 4.17 Hz with intermediate-layer alignment.
Spoken dialogue models typically start from text LLM backbones, yet reasoning often degrades when conditioning on speech instead of text. We attribute part of this modality gap to a temporal-granularity mismatch: speech tokens are temporally redundant and far longer than text under matched semantics, diluting per-token semantic density and weakening text-native reasoning dynamics. We study speech token design as a representation selection problem and sweep frame rates under a frozen LLM backbone with a fixed information rate. To make low frame rates feasible, we introduce factorized FSQ and a lightweight non-autoregressive audio LM head, scaling capacity to nearly 300\,bits/frame without sacrificing efficient prediction. With the bottleneck removed, we sweep frame rates (50$\rightarrow$2.08\,Hz) and alignment depth, and observe a consistent best regime for speech QA at 4.17\,Hz with intermediate-layer representation alignment.