Towards Efficient Vision State Space Models via Token Merging
This work addresses efficiency challenges for deploying SSMs in practical vision applications, though it appears incremental as it builds on existing token reduction methods.
The authors tackled the problem of computational inefficiency in State Space Models (SSMs) for vision tasks by proposing MaMe, a token-merging strategy tailored for SSMs, which achieved superior efficiency-performance trade-offs and maintained robustness under aggressive token reduction.
State Space Models (SSMs) have emerged as powerful architectures in computer vision, yet improving their computational efficiency remains crucial for practical and scalable deployment.While token reduction serves as an effective approach for model efficiency, applying it to SSMs requires careful consideration of their unique sequential modeling capabilities.In this work, we propose MaMe, a token-merging strategy tailored for SSM-based vision models.MaMe addresses two key challenges: quantifying token importance and preserving sequential properties. Our approach leverages the state transition parameter $\mathbfΔ$ as an informativeness measure and introduces strategic token arrangements to preserve sequential information flow.Extensive experiments demonstrate that MaMe achieves superior efficiency-performance trade-offs for both fine-tuned and off-the-shelf models. Particularly, our approach maintains robustness even under aggressive token reduction where existing methods undergo significant performance degradation.Beyond image classification, MaMe shows strong generalization capabilities across video and audio domains, establishing an effective approach for enhancing efficiency in diverse SSM applications.