HANDI: Hand-Centric Text-and-Image Conditioned Video Generation
This addresses the problem of generating realistic hand-centric action videos for applications like robotics and virtual reality, representing an incremental advance in video generation.
The paper tackles the challenge of generating videos with detailed hand motions in complex environments by introducing a diffusion-based method with automatic motion area generation and a hand refinement loss, achieving significant improvements in action clarity over state-of-the-art methods on augmented EpicKitchens and Ego4D datasets.
Despite the recent strides in video generation, state-of-the-art methods still struggle with elements of visual detail. One particularly challenging case is the class of videos in which the intricate motion of the hand coupled with a mostly stable and otherwise distracting environment is necessary to convey the execution of some complex action and its effects. To address these challenges, we introduce a new method for video generation that focuses on hand-centric actions. Our diffusion-based method incorporates two distinct innovations. First, we propose an automatic method to generate the motion area -- the region in the video in which the detailed activities occur -- guided by both the visual context and the action text prompt, rather than assuming this region can be provided manually as is now commonplace. Second, we introduce a critical Hand Refinement Loss to guide the diffusion model to focus on smooth and consistent hand poses. We evaluate our method on challenging augmented datasets based on EpicKitchens and Ego4D, demonstrating significant improvements over state-of-the-art methods in terms of action clarity, especially of the hand motion in the target region, across diverse environments and actions. Video results can be found in https://excitedbutter.github.io/project_page