LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders
This work addresses the critical challenge of adversarial robustness for computer vision systems, offering a principled solution to improve stability and performance trade-offs, though it appears incremental as it builds on existing fine-tuning strategies.
The paper tackled the problem of adversarial robustness in visual encoders by proposing LORE, an unsupervised adversarial fine-tuning framework that uses constrained optimization to balance robustness and clean data accuracy, resulting in significant improvements in zero-shot adversarial robustness with minimal degradation in clean data accuracy.
Visual encoders have become fundamental components in modern computer vision pipelines. However, ensuring robustness against adversarial perturbations remains a critical challenge. Recent efforts have explored both supervised and unsupervised adversarial fine-tuning strategies. We identify two key limitations in these approaches: (i) they often suffer from instability, especially during the early stages of fine-tuning, resulting in suboptimal convergence and degraded performance on clean data, and (ii) they exhibit a suboptimal trade-off between robustness and clean data accuracy, hindering the simultaneous optimization of both objectives. To overcome these challenges, we propose Lagrangian-Optimized Robust Embeddings (LORE), a novel unsupervised adversarial fine-tuning framework. LORE utilizes constrained optimization, which offers a principled approach to balancing competing goals, such as improving robustness while preserving nominal performance. By enforcing embedding-space proximity constraints, LORE effectively maintains clean data performance throughout adversarial fine-tuning. Extensive experiments show that LORE significantly improves zero-shot adversarial robustness with minimal degradation in clean data accuracy. Furthermore, we demonstrate the effectiveness of the adversarially fine-tuned CLIP image encoder in out-of-distribution generalization and enhancing the interpretability of image embeddings.