Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions
This addresses depth estimation robustness for applications like autonomous driving or robotics, but it is incremental as it builds on existing diffusion and self-distillation methods.
The paper tackles the problem of monocular depth estimation under challenging, out-of-distribution conditions by generating synthetic scenes with depth information using diffusion models and fine-tuning networks via self-distillation, achieving effectiveness and versatility as demonstrated in benchmarks.
We present a novel approach designed to address the complexities posed by challenging, out-of-distribution data in the single-image depth estimation task. Starting with images that facilitate depth prediction due to the absence of unfavorable factors, we systematically generate new, user-defined scenes with a comprehensive set of challenges and associated depth information. This is achieved by leveraging cutting-edge text-to-image diffusion models with depth-aware control, known for synthesizing high-quality image content from textual prompts while preserving the coherence of 3D structure between generated and source imagery. Subsequent fine-tuning of any monocular depth network is carried out through a self-distillation protocol that takes into account images generated using our strategy and its own depth predictions on simple, unchallenging scenes. Experiments on benchmarks tailored for our purposes demonstrate the effectiveness and versatility of our proposal.