Mirror3D: Depth Refinement for Mirror Surfaces
This addresses a specific error source in depth sensing and 3D reconstruction for applications like robotics and AR/VR, but is incremental as it builds on existing datasets and methods.
The paper tackles the problem of depth errors on mirror surfaces in 3D reconstruction by introducing Mirror3DNet, a module that refines raw or estimated depth using 3D mirror plane estimation from RGB and depth context, resulting in significant error mitigation across various input depth data.
Despite recent progress in depth sensing and 3D reconstruction, mirror surfaces are a significant source of errors. To address this problem, we create the Mirror3D dataset: a 3D mirror plane dataset based on three RGBD datasets (Matterport3D, NYUv2 and ScanNet) containing 7,011 mirror instance masks and 3D planes. We then develop Mirror3DNet: a module that refines raw sensor depth or estimated depth to correct errors on mirror surfaces. Our key idea is to estimate the 3D mirror plane based on RGB input and surrounding depth context, and use this estimate to directly regress mirror surface depth. Our experiments show that Mirror3DNet significantly mitigates errors from a variety of input depth data, including raw sensor depth and depth estimation or completion methods.