Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models
This addresses robustness issues in vision foundation models for 3D reasoning tasks, highlighting a critical but often overlooked problem in AI vision systems.
The paper analyzed viewpoint stability in nine vision foundation models, finding that they consistently encode accidental viewpoints but vary in handling out-of-distribution viewpoints, leading to object misclassifications and generalization gaps in 3D reasoning tasks.
In this paper, we analyze the viewpoint stability of foundational models - specifically, their sensitivity to changes in viewpoint- and define instability as significant feature variations resulting from minor changes in viewing angle, leading to generalization gaps in 3D reasoning tasks. We investigate nine foundational models, focusing on their responses to viewpoint changes, including the often-overlooked accidental viewpoints where specific camera orientations obscure an object's true 3D structure. Our methodology enables recognizing and classifying out-of-distribution (OOD), accidental, and stable viewpoints using feature representations alone, without accessing the actual images. Our findings indicate that while foundation models consistently encode accidental viewpoints, they vary in their interpretation of OOD viewpoints due to inherent biases, at times leading to object misclassifications based on geometric resemblance. Through quantitative and qualitative evaluations on three downstream tasks - classification, VQA, and 3D reconstruction - we illustrate the impact of viewpoint instability and underscore the importance of feature robustness across diverse viewing conditions.