DoubleTake: Geometry Guided Depth Estimation
This work addresses depth estimation for applications like augmented reality and path planning, offering an incremental improvement over prior methods.
The paper tackles depth estimation from posed RGB images by using historical 3D geometry predictions as an extra input to improve accuracy and regularization. It achieves state-of-the-art results in depth and 3D scene reconstruction, running at interactive speeds.
Estimating depth from a sequence of posed RGB images is a fundamental computer vision task, with applications in augmented reality, path planning etc. Prior work typically makes use of previous frames in a multi view stereo framework, relying on matching textures in a local neighborhood. In contrast, our model leverages historical predictions by giving the latest 3D geometry data as an extra input to our network. This self-generated geometric hint can encode information from areas of the scene not covered by the keyframes and it is more regularized when compared to individual predicted depth maps for previous frames. We introduce a Hint MLP which combines cost volume features with a hint of the prior geometry, rendered as a depth map from the current camera location, together with a measure of the confidence in the prior geometry. We demonstrate that our method, which can run at interactive speeds, achieves state-of-the-art estimates of depth and 3D scene reconstruction in both offline and incremental evaluation scenarios.