THOM: Generating Physically Plausible Hand-Object Meshes From Text
This addresses the need for physically plausible 3D hand-object interactions for dexterous robotic grasping and VR/AR content generation, representing a novel method for a known bottleneck.
The paper tackles the problem of generating 3D hand-object interaction meshes from text, which is challenging due to mesh extraction from text-generated Gaussians and physics-based optimization on erroneous meshes. The result is THOM, a training-free framework that surpasses state-of-the-art methods in text alignment, visual realism, and interaction plausibility.
The generation of 3D hand-object interactions (HOIs) from text is crucial for dexterous robotic grasping and VR/AR content generation, requiring both high visual fidelity and physical plausibility. Nevertheless, the ill-posed problem of mesh extraction from text-generated Gaussians, and physics-based optimization on the erroneous meshes pose challenges. To address these issues, we introduce THOM, a training-free framework that generates photorealistic, physically plausible 3D HOI meshes without the need for a template object mesh. THOM employs a two-stage pipeline, initially generating the hand and object Gaussians, followed by physics-based HOI optimization. Our new mesh extraction method and vertex-to-Gaussian mapping explicitly assign Gaussian elements to mesh vertices, allowing topology-aware regularization. Furthermore, we improve the physical plausibility of interactions by VLM-guided translation refinement and contact-aware optimization. Comprehensive experiments demonstrate that THOM consistently surpasses state-of-the-art methods in terms of text alignment, visual realism, and interaction plausibility.