StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs
This addresses a critical bottleneck for SpeechLLMs by enhancing noise robustness, though it is an incremental improvement over existing tokenizer methods.
The paper tackled the problem of semantic speech tokenizers being fragile to acoustic noise, which disrupts token sequences and burdens downstream language models, by introducing StableToken, a tokenizer that uses a multi-branch architecture with bit-wise voting to achieve stability, reducing Unit Edit Distance under noise and improving SpeechLLM robustness.
Prevalent semantic speech tokenizers, designed to capture linguistic content, are surprisingly fragile. We find they are not robust to meaning-irrelevant acoustic perturbations; even at high Signal-to-Noise Ratios (SNRs) where speech is perfectly intelligible, their output token sequences can change drastically, increasing the learning burden for downstream LLMs. This instability stems from two flaws: a brittle single-path quantization architecture and a distant training signal indifferent to intermediate token stability. To address this, we introduce StableToken, a tokenizer that achieves stability through a consensus-driven mechanism. Its multi-branch architecture processes audio in parallel, and these representations are merged via a powerful bit-wise voting mechanism to form a single, stable token sequence. StableToken sets a new state-of-the-art in token stability, drastically reducing Unit Edit Distance (UED) under diverse noise conditions. This foundational stability translates directly to downstream benefits, significantly improving the robustness of SpeechLLMs on a variety of tasks.