An Exploration of Mamba for Speech Self-Supervised Models
This work addresses the need for efficient and effective speech processing models, particularly for long sequences and real-time applications, though it is incremental as it adapts an existing method to a new domain.
The paper tackled the problem of applying Mamba, a linear-time selective state space model, to speech self-supervised learning as an alternative to Transformer-based architectures, resulting in significantly lower compute for long-context ASR fine-tuning, superior performance in streaming ASR, and competitive results on SUPERB benchmarks with higher-quality representations.
While Mamba has demonstrated strong performance in language modeling, its potential as a speech self-supervised (SSL) model remains underexplored, with prior studies limited to isolated tasks. To address this, we explore Mamba-based HuBERT models as alternatives to Transformer-based SSL architectures. Leveraging the linear-time Selective State Space, these models enable fine-tuning on long-context ASR with significantly lower compute. Moreover, they show superior performance when fine-tuned for streaming ASR. Beyond fine-tuning, these models show competitive performance on SUPERB probing benchmarks, particularly in causal settings. Our analysis shows that they yield higher-quality quantized representations and capture speaker-related features more distinctly than Transformer-based models. These findings highlight Mamba-based SSL as a promising and complementary direction for long-sequence modeling, real-time speech modeling, and speech unit extraction.