Test-Time Canonicalization by Foundation Models for Robust Perception
This addresses robustness issues in real-world perception for AI systems, offering a general and scalable approach that is incremental by building on existing foundation models.
The paper tackles the problem of perception robustness to diverse viewing conditions by proposing FOCAL, a test-time framework that transforms inputs into typical views using foundation model priors, resulting in significant robustness boosts across transformations like rotations and lighting shifts without retraining.
Perception in the real world requires robustness to diverse viewing conditions. Existing approaches often rely on specialized architectures or training with predefined data augmentations, limiting adaptability. Taking inspiration from mental rotation in human vision, we propose FOCAL, a test-time robustness framework that transforms the input into the most typical view. At inference time, FOCAL explores a set of transformed images and chooses the one with the highest likelihood under foundation model priors. This test-time optimization boosts robustness while requiring no retraining or architectural changes. Applied to models like CLIP and SAM, it significantly boosts robustness across a wide range of transformations, including 2D and 3D rotations, contrast and lighting shifts, and day-night changes. We also explore potential applications in active vision. By reframing invariance as a test-time optimization problem, FOCAL offers a general and scalable approach to robustness. Our code is available at: https://github.com/sutkarsh/focal.