Learning 3D Object Spatial Relationships from Pre-trained 2D Diffusion Models
This addresses the challenge of efficiently collecting 3D data for object spatial relationships, which is useful for robotics and computer vision applications, but it is incremental as it builds on existing diffusion models.
The paper tackles the problem of learning 3D spatial relationships between objects by using synthetically generated 3D samples from pre-trained 2D diffusion models, resulting in a method that demonstrates robustness across various relationships and applicability to tasks like 3D scene arrangement and human motion synthesis.
We present a method for learning 3D spatial relationships between object pairs, referred to as object-object spatial relationships (OOR), by leveraging synthetically generated 3D samples from pre-trained 2D diffusion models. We hypothesize that images synthesized by 2D diffusion models inherently capture realistic OOR cues, enabling efficient collection of a 3D dataset to learn OOR for various unbounded object categories. Our approach synthesizes diverse images that capture plausible OOR cues, which we then uplift into 3D samples. Leveraging our diverse collection of 3D samples for the object pairs, we train a score-based OOR diffusion model to learn the distribution of their relative spatial relationships. Additionally, we extend our pairwise OOR to multi-object OOR by enforcing consistency across pairwise relations and preventing object collisions. Extensive experiments demonstrate the robustness of our method across various object-object spatial relationships, along with its applicability to 3D scene arrangement tasks and human motion synthesis using our OOR diffusion model.