3DVNet: Multi-View Depth Prediction and Volumetric Refinement
This addresses the challenge of accurate 3D reconstruction from multiple views for applications in computer vision, though it appears incremental as it builds on existing MVS methods.
The paper tackles the problem of multi-view stereo depth prediction by introducing 3DVNet, which combines depth-based and volumetric approaches to achieve highly accurate predictions that agree on scene geometry, exceeding state-of-the-art accuracy on datasets like ScanNet, TUM-RGBD, and ICL-NUIM.
We present 3DVNet, a novel multi-view stereo (MVS) depth-prediction method that combines the advantages of previous depth-based and volumetric MVS approaches. Our key idea is the use of a 3D scene-modeling network that iteratively updates a set of coarse depth predictions, resulting in highly accurate predictions which agree on the underlying scene geometry. Unlike existing depth-prediction techniques, our method uses a volumetric 3D convolutional neural network (CNN) that operates in world space on all depth maps jointly. The network can therefore learn meaningful scene-level priors. Furthermore, unlike existing volumetric MVS techniques, our 3D CNN operates on a feature-augmented point cloud, allowing for effective aggregation of multi-view information and flexible iterative refinement of depth maps. Experimental results show our method exceeds state-of-the-art accuracy in both depth prediction and 3D reconstruction metrics on the ScanNet dataset, as well as a selection of scenes from the TUM-RGBD and ICL-NUIM datasets. This shows that our method is both effective and generalizes to new settings.