Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera
This work addresses the problem of metric depth estimation for applications using non-standard cameras, offering a significant performance boost but is incremental as it builds on existing depth foundation models.
The paper tackles the challenge of accurate metric depth estimation across diverse camera types, particularly fisheye and 360-degree cameras, by introducing Depth Any Camera (DAC), a zero-shot framework that extends perspective-trained models to handle varying fields of view without specialized training data, achieving up to 50% improvement in δ₁ accuracy on multiple datasets compared to prior models.
While recent depth foundation models exhibit strong zero-shot generalization, achieving accurate metric depth across diverse camera types-particularly those with large fields of view (FoV) such as fisheye and 360-degree cameras-remains a significant challenge. This paper presents Depth Any Camera (DAC), a powerful zero-shot metric depth estimation framework that extends a perspective-trained model to effectively handle cameras with varying FoVs. The framework is designed to ensure that all existing 3D data can be leveraged, regardless of the specific camera types used in new applications. Remarkably, DAC is trained exclusively on perspective images but generalizes seamlessly to fisheye and 360-degree cameras without the need for specialized training data. DAC employs Equi-Rectangular Projection (ERP) as a unified image representation, enabling consistent processing of images with diverse FoVs. Its core components include pitch-aware Image-to-ERP conversion with efficient online augmentation to simulate distorted ERP patches from undistorted inputs, FoV alignment operations to enable effective training across a wide range of FoVs, and multi-resolution data augmentation to further address resolution disparities between training and testing. DAC achieves state-of-the-art zero-shot metric depth estimation, improving $δ_1$ accuracy by up to 50% on multiple fisheye and 360-degree datasets compared to prior metric depth foundation models, demonstrating robust generalization across camera types.