Controllable 3D Placement of Objects with Scene-Aware Diffusion Models
This addresses the challenge of controllable object placement in image editing for applications like automotive design, though it is incremental as it builds on existing inpainting models.
The paper tackles the problem of precisely placing objects in 3D scenes with specific locations and orientations, using a scene-aware diffusion model that achieves high-quality placement by combining visual maps and coarse masks, as demonstrated in automotive settings with tasks measuring pose and location accuracy.
Image editing approaches have become more powerful and flexible with the advent of powerful text-conditioned generative models. However, placing objects in an environment with a precise location and orientation still remains a challenge, as this typically requires carefully crafted inpainting masks or prompts. In this work, we show that a carefully designed visual map, combined with coarse object masks, is sufficient for high quality object placement. We design a conditioning signal that resolves ambiguities, while being flexible enough to allow for changing of shapes or object orientations. By building on an inpainting model, we leave the background intact by design, in contrast to methods that model objects and background jointly. We demonstrate the effectiveness of our method in the automotive setting, where we compare different conditioning signals in novel object placement tasks. These tasks are designed to measure edit quality not only in terms of appearance, but also in terms of pose and location accuracy, including cases that require non-trivial shape changes. Lastly, we show that fine location control can be combined with appearance control to place existing objects in precise locations in a scene.