3D Photography using Context-aware Layered Depth Inpainting
This enables more realistic 3D photo generation from single images, benefiting applications in virtual reality and graphics, though it is incremental over existing layered depth approaches.
The paper tackles the problem of converting a single RGB-D image into a 3D photo for novel view synthesis, using a learning-based inpainting model to hallucinate color and depth in occluded regions, resulting in fewer artifacts compared to state-of-the-art methods.
We propose a method for converting a single RGB-D input image into a 3D photo - a multi-layer representation for novel view synthesis that contains hallucinated color and depth structures in regions occluded in the original view. We use a Layered Depth Image with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner. The resulting 3D photos can be efficiently rendered with motion parallax using standard graphics engines. We validate the effectiveness of our method on a wide range of challenging everyday scenes and show fewer artifacts compared with the state of the arts.