Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation
This work addresses the efficiency and interpretability of state-space models for neural sequence modeling, though it is incremental as it builds directly on Mamba2.
The authors tackled the lack of structured utilization in Mamba2's multi-head recurrence by proposing HADES, a Graph Signal Processing-inspired framework that reinterprets it as an adaptive filter bank, achieving comparable performance across benchmarks while using only 58.9% of the original parameters.
State-space models (SSMs) offer efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, uses selective input gating and a multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called Hierarchical ADaptive filter bank for Efficient SSMs (HADES), a Graph Signal Processing (GSP)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter Î. HADES achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only 58.9% of the original parameters. In this regard, HADES bridges GSP and neural sequence modeling, enabling efficient, hierarchical, and interpretable filtering within state-space models.