Toward A Better Understanding of Monocular Depth Evaluation
This work addresses the lack of standardization in evaluating monocular depth estimation, which is an important task in computer vision, but it is incremental as it builds on existing metrics rather than proposing a fundamentally new approach.
The paper tackles the problem of evaluating monocular depth estimation methods by analyzing existing metrics' sensitivity to ground truth perturbations and comparing them to human judgment. It reveals that current metrics are severely under-sensitive to curvature perturbations and introduces a new metric based on relative surface normals, along with visualization tools and a method for creating composite metrics with better human alignment.
Monocular depth estimation is an important task with rapid progress, but how to evaluate it is not fully resolved, as evidenced by a lack of standardization in existing literature and a large selection of evaluation metrics whose trade-offs and behaviors are not fully understood. This paper contributes a novel, quantitative analysis of existing metrics in terms of their sensitivity to various types of perturbations of ground truth, emphasizing comparison to human judgment. Our analysis reveals that existing metrics are severely under-sensitive to curvature perturbation such as making smooth surfaces bumpy. To remedy this, we introduce a new metric based on relative surface normals, along with new depth visualization tools and a principled method to create composite metrics with better human alignment. Code and data are available at: https://github.com/princeton-vl/evalmde.