NL2Contact: Natural Language Guided 3D Hand-Object Contact Modeling with Diffusion Model
This addresses the need for controllable contact modeling in 3D hand-object reconstruction, enabling text-guided refinement and generation of human grasps, though it is incremental as it builds on existing diffusion models and dataset creation methods.
The paper tackles the problem of generating controllable 3D hand-object contacts from natural language descriptions, proposing NL2Contact with a diffusion model and introducing the ContactDescribe dataset, achieving realistic and faithful contact modeling for applications like grasp pose optimization and novel grasp generation.
Modeling the physical contacts between the hand and object is standard for refining inaccurate hand poses and generating novel human grasp in 3D hand-object reconstruction. However, existing methods rely on geometric constraints that cannot be specified or controlled. This paper introduces a novel task of controllable 3D hand-object contact modeling with natural language descriptions. Challenges include i) the complexity of cross-modal modeling from language to contact, and ii) a lack of descriptive text for contact patterns. To address these issues, we propose NL2Contact, a model that generates controllable contacts by leveraging staged diffusion models. Given a language description of the hand and contact, NL2Contact generates realistic and faithful 3D hand-object contacts. To train the model, we build \textit{ContactDescribe}, the first dataset with hand-centered contact descriptions. It contains multi-level and diverse descriptions generated by large language models based on carefully designed prompts (e.g., grasp action, grasp type, contact location, free finger status). We show applications of our model to grasp pose optimization and novel human grasp generation, both based on a textual contact description.