Res$^2$CLIP: Few-Shot Generalist Anomaly Detection with Residual-to-Residual Alignment
This work addresses the challenge of few-shot anomaly detection for generalist models, enabling adaptation to novel categories without retraining, which is critical for real-world applications with scarce samples and rapidly changing categories.
Res^2CLIP introduces a residual-to-residual alignment framework for few-shot generalist anomaly detection, shifting multimodal alignment into a unified residual space to resolve cross-granularity mismatch and cross-category generalization degradation. Experiments show effectiveness across multiple datasets.
Few-shot Generalist Anomaly Detection requires models to generalize to novel categories without retraining, posing significant challenges in real-world scenarios with scarce samples and rapidly changing categories. Existing CLIP-based methods face two major challenges: coarse-grained unified text prompts struggle to adapt to fine-grained foreground-background differences, causing cross-granularity mismatch; and fine-tuning on auxiliary datasets disrupts CLIP's inherent open-world generalization due to domain shift, leading to cross-category generalization degradation. To address these, we propose to shift multimodal alignment entirely into a unified residual space, where residual representations naturally eliminate fine-grained normal feature differences across regions and class-specific biases, simultaneously resolving both problems. Based on this insight, Res$^2$CLIP, the first residual-to-residual alignment framework that symmetrically bridges visual and text modalities within CLIP's residual space, is designed. The framework is developed from a residual perspective into three branches: a text prompt-based branch, a visual prompt-based branch, and a novel residual-to-residual alignment branch. All learnable optimizations are constrained within the residual domain, and the residual alignment optimization objectives are designed to force the model to focus on relative anomaly deviations rather than optimizing class-specific features. Experiments on multiple datasets demonstrate the effectiveness of our architecture. The code is available at https://github.com/hito2448/Res2CLIP.