Alligat0R: Pre-Training Through Co-Visibility Segmentation for Relative Camera Pose Regression
This work addresses a bottleneck in computer vision pre-training for tasks like 3D reconstruction and pose regression, offering a more flexible and interpretable method that can handle diverse overlap scenarios, though it is incremental as it builds on prior cross-view learning techniques.
The paper tackles the limitation of existing pre-training methods like CroCo, which require substantial overlap between image pairs, by introducing Alligat0R, a novel pre-training approach that reformulates cross-view learning as co-visibility segmentation, enabling use of image pairs with any overlap and significantly outperforming CroCo in relative pose regression, particularly in scenarios with limited overlap.
Pre-training techniques have greatly advanced computer vision, with CroCo's cross-view completion approach yielding impressive results in tasks like 3D reconstruction and pose regression. However, this method requires substantial overlap between training pairs, limiting its effectiveness. We introduce Alligat0R, a novel pre-training approach that reformulates cross-view learning as a co-visibility segmentation task. Our method predicts whether each pixel in one image is co-visible in the second image, occluded, or outside the field of view (FOV), enabling the use of image pairs with any degree of overlap and providing interpretable predictions. To support this, we present Cub3, a large-scale dataset with 2.5 million image pairs and dense co-visibility annotations derived from the nuScenes dataset. This dataset includes diverse scenarios with varying degrees of overlap. The experiments show that Alligat0R significantly outperforms CroCo in relative pose regression, especially in scenarios with limited overlap. Alligat0R and Cub3 will be made publicly available.