Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba
This addresses the need for efficient fine-tuning of large State Space Models like Mamba for researchers and practitioners, though it is incremental as it builds on existing PEFT methods.
The paper tackled the problem of adapting pre-trained Mamba models to downstream tasks efficiently by proposing Memba, a parameter-efficient fine-tuning method that uses bio-inspired gating mechanisms, resulting in substantial improvements over existing methods in language and vision tasks.
State Space Models (SSMs) have emerged as powerful alternatives to attention-based Transformers, with Mamba demonstrating impressive efficiency and scalability. As these models grow increasingly larger, the need for Parameter-Efficient Fine-Tuning (PEFT) methods becomes critical to adapt pre-trained Mamba to downstream tasks without prohibitive computational costs. However, previous approaches simply apply traditional Transformer-tailored PEFT methods without addressing the unique temporal processing dynamics of SSMs. To address this limitation, we propose Memba, a membrane-driven PEFT approach specifically designed for Mamba. Memba introduces Leaky Integrate Membrane (LIM) neurons as bio-inspired gating mechanisms that naturally accumulate membrane potentials over time, enhancing selective information retention. By strategically combining LIM neurons with Low-Rank Adaptations (LoRA) and cross-layer membrane transfer, our approach significantly improves Mamba's temporal modeling capabilities. Extensive experiments across language and vision tasks demonstrate that Memba achieves substantial improvements over existing PEFT methods. The code is available at https://github.com/Intelligent-Computing-Lab-Yale/Memba.