Near, far: Patch-ordering enhances vision foundation models' scene understanding
This work addresses the need for better dense feature encoders in computer vision, offering a novel method that achieves state-of-the-art results across multiple datasets, though it is incremental as it builds on existing pretrained representations.
The paper tackles the problem of improving vision foundation models' scene understanding by introducing NeCo, a self-supervised training loss that enforces patch-level nearest neighbor consistency, resulting in superior performance such as +5.5% and +6% gains for non-parametric in-context semantic segmentation on ADE20k and Pascal VOC.
We introduce NeCo: Patch Neighbor Consistency, a novel self-supervised training loss that enforces patch-level nearest neighbor consistency across a student and teacher model. Compared to contrastive approaches that only yield binary learning signals, i.e., 'attract' and 'repel', this approach benefits from the more fine-grained learning signal of sorting spatially dense features relative to reference patches. Our method leverages differentiable sorting applied on top of pretrained representations, such as DINOv2-registers to bootstrap the learning signal and further improve upon them. This dense post-pretraining leads to superior performance across various models and datasets, despite requiring only 19 hours on a single GPU. This method generates high-quality dense feature encoders and establishes several new state-of-the-art results such as +5.5% and +6% for non-parametric in-context semantic segmentation on ADE20k and Pascal VOC, +7.2% and +5.7% for linear segmentation evaluations on COCO-Things and -Stuff and improvements in the 3D understanding of multi-view consistency on SPair-71k, by more than 1.5%.