Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modeling
This addresses the need for efficient and generalizable 3D scene modeling for applications like virtual reality or robotics, offering a zero-shot approach that avoids costly data requirements.
The paper tackles the problem of reconstructing structured 3D indoor scenes from single-view RGB images without training or annotations, achieving significant performance improvements over baselines on synthetic and real-world data.
Reconstructing structured 3D scenes from RGB images using CAD objects unlocks efficient and compact scene representations that maintain compositionality and interactability. Existing works propose training-heavy methods relying on either expensive yet inaccurate real-world annotations or controllable yet monotonous synthetic data that do not generalize well to unseen objects or domains. We present Diorama, the first zero-shot open-world system that holistically models 3D scenes from single-view RGB observations without requiring end-to-end training or human annotations. We show the feasibility of our approach by decomposing the problem into subtasks and introduce robust, generalizable solutions to each: architecture reconstruction, 3D shape retrieval, object pose estimation, and scene layout optimization. We evaluate our system on both synthetic and real-world data to show we significantly outperform baselines from prior work. We also demonstrate generalization to internet images and the text-to-scene task.