Φeat: Physically-Grounded Feature Representation
This addresses the need for physics-aware perception in vision and graphics, offering a novel unsupervised approach for tasks requiring explicit physical reasoning, though it builds on existing data and methods in an incremental way.
The paper tackles the problem of self-supervised visual features entangling semantics with physical factors like geometry and illumination, hindering physical reasoning tasks, and introduces Φeat, a physically-grounded backbone that learns material identity through contrastive pretraining, showing it captures physically-grounded structure beyond semantic grouping.
Foundation models have emerged as effective backbones for many vision tasks. However, current self-supervised features entangle high-level semantics with low-level physical factors, such as geometry and illumination, hindering their use in tasks requiring explicit physical reasoning. In this paper, we introduce $Φ$eat, a novel physically-grounded visual backbone that encourages a representation sensitive to material identity, including reflectance cues and geometric mesostructure. Our key idea is to employ a pretraining strategy that contrasts spatial crops and physical augmentations of the same material under varying shapes and lighting conditions. While similar data have been used in high-end supervised tasks such as intrinsic decomposition or material estimation, we demonstrate that a pure self-supervised training strategy, without explicit labels, already provides a strong prior for tasks requiring robust features invariant to external physical factors. We evaluate the learned representations through feature similarity analysis and material selection, showing that $Φ$eat captures physically-grounded structure beyond semantic grouping. These findings highlight the promise of unsupervised physical feature learning as a foundation for physics-aware perception in vision and graphics. These findings highlight the promise of unsupervised physical feature learning as a foundation for physics-aware perception in vision and graphics.