Muskie: Multi-view Masked Image Modeling for 3D Vision Pre-training
This addresses the need for better multi-view consistency in 3D vision tasks, offering a novel pre-training approach that enhances geometric understanding without 3D supervision, though it is incremental as it builds on masked image modeling techniques.
The paper tackles the problem of limited multi-view consistency in 3D vision by introducing Muskie, a multi-view masked image modeling backbone that reconstructs masked content using geometric correspondences from other views, achieving higher multi-view correspondence accuracy than state-of-the-art frame-wise backbones like DINO and improving performance on downstream tasks such as camera pose estimation and pointmap reconstruction.
We present Muskie, a native multi-view vision backbone designed for 3D vision tasks. Unlike existing models, which are frame-wise and exhibit limited multi-view consistency, Muskie is designed to process multiple views simultaneously and introduce multi-view consistency in pre-training stage. Muskie is trained to reconstruct heavily masked content in one view by finding and utilizing geometric correspondences from other views. Through this pretext task and our proposed aggressive masking strategy, the model implicitly to learn view-invariant features and develop strong geometric understanding without any 3D supervision. Compared with state-of-the-art frame-wise backbones such as DINO, Muskie achieves higher multi-view correspondence accuracy. Furthermore, we demonstrate that using Muskie as a backbone consistently enhances performance on downstream 3D tasks, including camera pose estimation and pointmap reconstruction. Codes are publicly available at https://leo-frank.github.io/Muskie/