Hoi3DGen: Generating High-Quality Human-Object-Interactions in 3D
This work addresses a crucial need for high-quality 3D human-object interaction generation in applications like AR, XR, and gaming, representing a strong specific gain rather than an incremental improvement.
The paper tackles the problem of generating 3D human-object interactions from text, which often suffers from issues like the Janus problem and poor text fidelity due to data scarcity, and introduces Hoi3DGen, a framework that achieves orders-of-magnitude improvements in interaction fidelity, surpassing baselines by 4-15x in text consistency and 3-7x in 3D model quality.
Modeling and generating 3D human-object interactions from text is crucial for applications in AR, XR, and gaming. Existing approaches often rely on score distillation from text-to-image models, but their results suffer from the Janus problem and do not follow text prompts faithfully due to the scarcity of high-quality interaction data. We introduce Hoi3DGen, a framework that generates high-quality textured meshes of human-object interaction that follow the input interaction descriptions precisely. We first curate realistic and high-quality interaction data leveraging multimodal large language models, and then create a full text-to-3D pipeline, which achieves orders-of-magnitude improvements in interaction fidelity. Our method surpasses baselines by 4-15x in text consistency and 3-7x in 3D model quality, exhibiting strong generalization to diverse categories and interaction types, while maintaining high-quality 3D generation.