Language-Based Depth Hints for Monocular Depth Estimation
This work addresses depth estimation ambiguity for computer vision applications, but it is incremental as it builds on existing methods with a novel language-based hint.
The paper tackled the inherent ambiguity in monocular depth estimation by using natural language as an explicit prior to predict depth distributions of objects, demonstrating improved performance on the NYUD2 dataset compared to baselines.
Monocular depth estimation (MDE) is inherently ambiguous, as a given image may result from many different 3D scenes and vice versa. To resolve this ambiguity, an MDE system must make assumptions about the most likely 3D scenes for a given input. These assumptions can be either explicit or implicit. In this work, we demonstrate the use of natural language as a source of an explicit prior about the structure of the world. The assumption is made that human language encodes the likely distribution in depth-space of various objects. We first show that a language model encodes this implicit bias during training, and that it can be extracted using a very simple learned approach. We then show that this prediction can be provided as an explicit source of assumption to an MDE system, using an off-the-shelf instance segmentation model that provides the labels used as the input to the language model. We demonstrate the performance of our method on the NYUD2 dataset, showing improvement compared to the baseline and to random controls.