TRUST: Test-Time Refinement using Uncertainty-Guided SSM Traverses
This addresses a robustness issue for vision tasks using SSMs, but it is incremental as it builds on existing test-time adaptation techniques.
The paper tackles the problem of State Space Models (SSMs) degrading in performance under distribution shifts by proposing TRUST, a test-time adaptation method that uses uncertainty-guided traversals and pseudo-labels to update parameters, resulting in improved robustness and outperforming existing methods across seven benchmarks.
State Space Models (SSMs) have emerged as efficient alternatives to Vision Transformers (ViTs), with VMamba standing out as a pioneering architecture designed for vision tasks. However, their generalization performance degrades significantly under distribution shifts. To address this limitation, we propose TRUST (Test-Time Refinement using Uncertainty-Guided SSM Traverses), a novel test-time adaptation (TTA) method that leverages diverse traversal permutations to generate multiple causal perspectives of the input image. Model predictions serve as pseudo-labels to guide updates of the Mamba-specific parameters, and the adapted weights are averaged to integrate the learned information across traversal scans. Altogether, TRUST is the first approach that explicitly leverages the unique architectural properties of SSMs for adaptation. Experiments on seven benchmarks show that TRUST consistently improves robustness and outperforms existing TTA methods.