MapAnything: Universal Feed-Forward Metric 3D Reconstruction
This provides a universal backbone for 3D reconstruction that could simplify pipelines for computer vision researchers and practitioners, though it appears incremental as it builds on existing transformer and factored representation concepts.
The authors tackled the problem of diverse 3D vision tasks requiring separate specialized models by introducing MapAnything, a unified transformer-based feed-forward model that directly regresses metric 3D scene geometry and cameras from images with optional geometric inputs, which outperforms or matches specialist models while enabling more efficient joint training.
We introduce MapAnything, a unified transformer-based feed-forward model that ingests one or more images along with optional geometric inputs such as camera intrinsics, poses, depth, or partial reconstructions, and then directly regresses the metric 3D scene geometry and cameras. MapAnything leverages a factored representation of multi-view scene geometry, i.e., a collection of depth maps, local ray maps, camera poses, and a metric scale factor that effectively upgrades local reconstructions into a globally consistent metric frame. Standardizing the supervision and training across diverse datasets, along with flexible input augmentation, enables MapAnything to address a broad range of 3D vision tasks in a single feed-forward pass, including uncalibrated structure-from-motion, calibrated multi-view stereo, monocular depth estimation, camera localization, depth completion, and more. We provide extensive experimental analyses and model ablations demonstrating that MapAnything outperforms or matches specialist feed-forward models while offering more efficient joint training behavior, thus paving the way toward a universal 3D reconstruction backbone.