LPA3D: 3D Room-Level Scene Generation from In-the-Wild Images
This work addresses a crucial bottleneck for applications in VR, AR, and robotics by enabling scene generation from single images without additional inputs like multiple views or depth, though it builds incrementally on NeRF-based generative techniques.
The paper tackles the problem of generating realistic 3D room-level indoor scenes from single in-the-wild RGB images, which is challenging due to the need for camera pose estimation in NeRF-based methods. It introduces LPA-GAN, a novel approach that redefines global poses using a Local-Pose-Alignment framework and co-optimizes pose prediction with scene generation, achieving superior view-to-view consistency and semantic normality compared to existing methods.
Generating realistic, room-level indoor scenes with semantically plausible and detailed appearances from in-the-wild images is crucial for various applications in VR, AR, and robotics. The success of NeRF-based generative methods indicates a promising direction to address this challenge. However, unlike their success at the object level, existing scene-level generative methods require additional information, such as multiple views, depth images, or semantic guidance, rather than relying solely on RGB images. This is because NeRF-based methods necessitate prior knowledge of camera poses, which is challenging to approximate for indoor scenes due to the complexity of defining alignment and the difficulty of globally estimating poses from a single image, given the unseen parts behind the camera. To address this challenge, we redefine global poses within the framework of Local-Pose-Alignment (LPA) -- an anchor-based multi-local-coordinate system that uses a selected number of anchors as the roots of these coordinates. Building on this foundation, we introduce LPA-GAN, a novel NeRF-based generative approach that incorporates specific modifications to estimate the priors of camera poses under LPA. It also co-optimizes the pose predictor and scene generation processes. Our ablation study and comparisons with straightforward extensions of NeRF-based object generative methods demonstrate the effectiveness of our approach. Furthermore, visual comparisons with other techniques reveal that our method achieves superior view-to-view consistency and semantic normality.