MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Models
This work addresses the problem of adapting sequence models to visual data for researchers and practitioners in computer vision, offering a novel architecture that improves performance across multiple benchmarks, though it appears incremental relative to prior SSM adaptations.
The paper tackles the challenge of extending State Space Models (SSMs) like Mamba to computer vision by addressing spatial redundancy and complex 2D dependencies in images, proposing MFil-Mamba with a multi-filter scanning backbone and adaptive weighting. It achieves superior performance, e.g., 83.2% top-1 accuracy on ImageNet-1K and 47.3% box AP on MS COCO.
State Space Models (SSMs), especially recent Mamba architecture, have achieved remarkable success in sequence modeling tasks. However, extending SSMs to computer vision remains challenging due to the non-sequential structure of visual data and its complex 2D spatial dependencies. Although several early studies have explored adapting selective SSMs for vision applications, most approaches primarily depend on employing various traversal strategies over the same input. This introduces redundancy and distorts the intricate spatial relationships within images. To address these challenges, we propose MFil-Mamba, a novel visual state space architecture built on a multi-filter scanning backbone. Unlike fixed multi-directional traversal methods, our design enables each scan to capture unique and contextually relevant spatial information while minimizing redundancy. Furthermore, we incorporate an adaptive weighting mechanism to effectively fuse outputs from multiple scans in addition to architectural enhancements. MFil-Mamba achieves superior performance over existing state-of-the-art models across various benchmarks that include image classification, object detection, instance segmentation, and semantic segmentation. For example, our tiny variant attains 83.2% top-1 accuracy on ImageNet-1K, 47.3% box AP and 42.7% mask AP on MS COCO, and 48.5% mIoU on the ADE20K dataset. Code and models are available at https://github.com/puskal-khadka/MFil-Mamba.