Spectral State Space Model for Rotation-Invariant Visual Representation Learning
This work addresses a domain-specific problem for computer vision researchers by improving SSMs for rotation-invariant representation learning, though it is incremental as it builds on existing SSM frameworks.
The paper tackled the limitations of State Space Models (SSMs) in vision, which fail to capture non-adjacent patch relationships and are sensitive to rotations, by introducing Spectral VMamba, a method that uses spectral decomposition and a Rotational Feature Normalizer to achieve rotation invariance and improve performance, outperforming leading SSM models like VMamba in classification tasks with similar runtime efficiency.
State Space Models (SSMs) have recently emerged as an alternative to Vision Transformers (ViTs) due to their unique ability of modeling global relationships with linear complexity. SSMs are specifically designed to capture spatially proximate relationships of image patches. However, they fail to identify relationships between conceptually related yet not adjacent patches. This limitation arises from the non-causal nature of image data, which lacks inherent directional relationships. Additionally, current vision-based SSMs are highly sensitive to transformations such as rotation. Their predefined scanning directions depend on the original image orientation, which can cause the model to produce inconsistent patch-processing sequences after rotation. To address these limitations, we introduce Spectral VMamba, a novel approach that effectively captures the global structure within an image by leveraging spectral information derived from the graph Laplacian of image patches. Through spectral decomposition, our approach encodes patch relationships independently of image orientation, achieving rotation invariance with the aid of our Rotational Feature Normalizer (RFN) module. Our experiments on classification tasks show that Spectral VMamba outperforms the leading SSM models in vision, such as VMamba, while maintaining invariance to rotations and a providing a similar runtime efficiency.