Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
This work addresses a bottleneck in scaling SSMs for long sequence modeling, offering an incremental improvement in efficiency for language modeling tasks.
The paper tackles the challenge of efficiently scaling State Space Models (SSMs) with Mixture-of-Experts (MoE) by introducing Routing Mamba (RoM), which uses sparse mixtures of linear projection experts to achieve language modeling performance equivalent to a dense Mamba model with over 2.3x more active parameters and a 23% FLOPS saving.
Linear State Space Models (SSMs) offer remarkable performance gains in efficient sequence modeling, with constant inference-time computation and memory complexity. Recent advances, such as Mamba, further enhance SSMs with input-dependent gating and hardware-aware implementations, positioning them as strong alternatives to Transformers for long sequence modeling. However, efficiently scaling the expressive power of SSMs, particularly with Mixture of Experts (MoE), remains challenging, as naive integration attempts often falter or degrade performance. In this work, we introduce Routing Mamba (RoM), a novel approach that scales SSM parameters using sparse mixtures of linear projection experts. By sharing routing decisions between projection layers and lightweight sub-modules within Mamba across experts, RoM leverages synergies among linear projection experts for effective and efficient sparse scaling of Mamba layers. At a scale of 1.3B active parameters (10B total) and 16K training sequence length, RoM achieves language modeling performance equivalent to a dense Mamba model requiring over 2.3x more active parameters, and demonstrates consistent perplexity across context lengths. Experimental results further show RoM effectively scales hybrid language models, yielding a 23% FLOPS saving compared to dense Mamba scaling for similar performance.