Pano3DComposer: Feed-Forward Compositional 3D Scene Generation from Single Panoramic Image
This work is significant for researchers and practitioners in 3D scene generation, offering a more efficient and complete approach for creating 360-degree environments, especially for image/text-to-3D tasks.
The paper introduces Pano3DComposer, a feed-forward framework that generates 3D scenes from single panoramic images, addressing limitations of iterative optimization and reliance on limited field-of-view images. It achieves superior geometric accuracy and can generate a high-fidelity 3D scene in approximately 20 seconds on an RTX 4090 GPU.
Current compositional image-to-3D scene generation approaches construct 3D scenes by time-consuming iterative layout optimization or inflexible joint object-layout generation. Moreover, most methods rely on limited field-of-view perspective images, hindering the creation of complete 360-degree environments. To address these limitations, we design Pano3DComposer, an efficient feed-forward framework for panoramic images. To decouple object generation from layout estimation, we propose a plug-and-play Object-World Transformation Predictor. This module converts the 3D objects generated by off-the-shelf image-to-3D models from local to world coordinates. To achieve this, we adapt the VGGT architecture to Alignment-VGGT by using target object crop, multi-view object renderings and camera parameters to predict the transformation. The predictor is trained using pseudo-geometric supervision to address the shape discrepancy between generated and ground-truth objects. For input images from unseen domains, we further introduce a Coarse-to-Fine (C2F) alignment mechanism for Pano3DComposer that iteratively refines geometric consistency with feedback of scene rendering. Our method achieves superior geometric accuracy for image/text-to-3D tasks on synthetic and real-world datasets. It can generate a high-fidelity 3D scene in approximately 20 seconds on an RTX 4090 GPU. Project page: https://qiuzidian.github.io/pano3dcomposer-page/.