BenchDepth: Are We on the Right Way to Evaluate Depth Foundation Models?
This work addresses a critical issue for the computer vision community by offering a standardized evaluation framework that could improve research and development in depth estimation, though it is incremental as it builds on existing methods.
The authors tackled the problem of inconsistent evaluation protocols for depth foundation models by proposing BenchDepth, a benchmark that assesses models through five downstream proxy tasks, and found that it provides a more practical and fair comparison than traditional alignment-based metrics.
Depth estimation is a fundamental task in computer vision with diverse applications. Recent advancements in deep learning have led to powerful depth foundation models (DFMs), yet their evaluation remains challenging due to inconsistencies in existing protocols. Traditional benchmarks rely on alignment-based metrics that introduce biases, favor certain depth representations, and complicate fair comparisons. In this work, we propose BenchDepth, a new benchmark that evaluates DFMs through five carefully selected downstream proxy tasks: depth completion, stereo matching, monocular feed-forward 3D scene reconstruction, SLAM, and vision-language spatial understanding. Unlike conventional evaluation protocols, our approach assesses DFMs based on their practical utility in real-world applications, bypassing problematic alignment procedures. We benchmark eight state-of-the-art DFMs and provide an in-depth analysis of key findings and observations. We hope our work sparks further discussion in the community on best practices for depth model evaluation and paves the way for future research and advancements in depth estimation.