Multi-View Foundation Models
This addresses the issue of feature inconsistency in multi-view computer vision tasks, offering a direct image-space solution for applications like surface normal estimation and segmentation, though it is incremental as it builds on existing foundation models.
The paper tackles the problem of inconsistent feature representations across multiple views of a 3D scene in foundation models, proposing a method to convert them into multi-view models that enhance feature consistency, with quantitative experiments showing considerable improvements in feature matching.
Foundation models are vital tools in various Computer Vision applications. They take as input a single RGB image and output a deep feature representation that is useful for various applications. However, in case we have multiple views of the same 3D scene, they operate on each image independently and do not always produce consistent features for the same 3D point. We propose a way to convert a Foundation Model into a Multi-View Foundation Model. Such a model takes as input a set of images and outputs a feature map for each image such that the features of corresponding points are as consistent as possible. This approach bypasses the need to build a consistent 3D model of the features and allows direct manipulation in the image space. Specifically, we show how to augment Transformers-based foundation models (i.e., DINO, SAM, CLIP) with intermediate 3D-aware attention layers that help match features across different views. As leading examples, we show surface normal estimation and multi-view segmentation tasks. Quantitative experiments show that our method improves feature matching considerably compared to current foundation models.