SAMwave: Wavelet-Driven Feature Enrichment for Effective Adaptation of Segment Anything Model
This addresses the challenge of adapting foundation models like SAM for specific, complex vision tasks, though it is incremental as it builds on existing adapter-based fine-tuning strategies.
The paper tackles the problem of adapting the Segment Anything Model (SAM) to complex tasks where it suffers performance degradation, proposing SAMwave, which uses wavelet transforms to extract multi-scale high-frequency features, and shows it significantly outperforms existing adaptation methods across four low-level vision tasks.
The emergence of large foundation models has propelled significant advances in various domains. The Segment Anything Model (SAM), a leading model for image segmentation, exemplifies these advances, outperforming traditional methods. However, such foundation models often suffer from performance degradation when applied to complex tasks for which they are not trained. Existing methods typically employ adapter-based fine-tuning strategies to adapt SAM for tasks and leverage high-frequency features extracted from the Fourier domain. However, Our analysis reveals that these approaches offer limited benefits due to constraints in their feature extraction techniques. To overcome this, we propose \textbf{\textit{SAMwave}}, a novel and interpretable approach that utilizes the wavelet transform to extract richer, multi-scale high-frequency features from input data. Extending this, we introduce complex-valued adapters capable of capturing complex-valued spatial-frequency information via complex wavelet transforms. By adaptively integrating these wavelet coefficients, SAMwave enables SAM's encoder to capture information more relevant for dense prediction. Empirical evaluations on four challenging low-level vision tasks demonstrate that SAMwave significantly outperforms existing adaptation methods. This superior performance is consistent across both the SAM and SAM2 backbones and holds for both real and complex-valued adapter variants, highlighting the efficiency, flexibility, and interpretability of our proposed method for adapting segment anything models.