POCI-Diff: Position Objects Consistently and Interactively with 3D-Layout Guided Diffusion
This work solves the problem of interactive 3D layout-guided image generation and editing for users in computer vision and graphics, offering a novel framework that improves over existing incremental approaches.
The paper tackles the problem of generating and editing images from text with consistent 3D layout control, addressing distortions and inconsistencies in prior methods, and demonstrates that POCI-Diff outperforms state-of-the-art methods in visual fidelity and layout adherence while eliminating geometric artifacts.
We propose a diffusion-based approach for Text-to-Image (T2I) generation with consistent and interactive 3D layout control and editing. While prior methods improve spatial adherence using 2D cues or iterative copy-warp-paste strategies, they often distort object geometry and fail to preserve consistency across edits. To address these limitations, we introduce a framework for Positioning Objects Consistently and Interactively (POCI-Diff), a novel formulation for jointly enforcing 3D geometric constraints and instance-level semantic binding within a unified diffusion process. Our method enables explicit per-object semantic control by binding individual text descriptions to specific 3D bounding boxes through Blended Latent Diffusion, allowing one-shot synthesis of complex multi-object scenes. We further propose a warping-free generative editing pipeline that supports object insertion, removal, and transformation via regeneration rather than pixel deformation. To preserve object identity and consistency across edits, we condition the diffusion process on reference images using IP-Adapter, enabling coherent object appearance throughout interactive 3D editing while maintaining global scene coherence. Experimental results demonstrate that POCI-Diff produces high-quality images consistent with the specified 3D layouts and edits, outperforming state-of-the-art methods in both visual fidelity and layout adherence while eliminating warping-induced geometric artifacts.