iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformer
This addresses the need for more natural digital human videos in applications like education and e-commerce, though it appears incremental as it builds on existing video diffusion transformer models.
The paper tackles the problem of generating realistic Hand-Object Interaction (HOI) reenactments in videos, which is challenging due to occlusion, object variations, and physical interaction requirements, and presents iDiT-HOI, a framework that outperforms existing methods in real-world scenes with enhanced realism and seamless interactions.
Digital human video generation is gaining traction in fields like education and e-commerce, driven by advancements in head-body animation and lip-syncing technologies. However, realistic Hand-Object Interaction (HOI) - the complex dynamics between human hands and objects - continues to pose challenges. Generating natural and believable HOI reenactments is difficult due to issues such as occlusion between hands and objects, variations in object shapes and orientations, and the necessity for precise physical interactions, and importantly, the ability to generalize to unseen humans and objects. This paper presents a novel framework iDiT-HOI that enables in-the-wild HOI reenactment generation. Specifically, we propose a unified inpainting-based token process method, called Inp-TPU, with a two-stage video diffusion transformer (DiT) model. The first stage generates a key frame by inserting the designated object into the hand region, providing a reference for subsequent frames. The second stage ensures temporal coherence and fluidity in hand-object interactions. The key contribution of our method is to reuse the pretrained model's context perception capabilities without introducing additional parameters, enabling strong generalization to unseen objects and scenarios, and our proposed paradigm naturally supports long video generation. Comprehensive evaluations demonstrate that our approach outperforms existing methods, particularly in challenging real-world scenes, offering enhanced realism and more seamless hand-object interactions.