Mask2IV: Interaction-Centric Video Generation via Mask Trajectories
This work addresses the challenge of modeling complex interactions in video generation for robot learning and manipulation, though it is incremental in improving controllability and reducing annotation requirements.
The paper tackles the problem of generating interaction-centric videos for embodied intelligence by introducing Mask2IV, a two-stage framework that predicts motion trajectories and then generates videos, eliminating the need for dense mask annotations. It achieves superior visual realism and controllability compared to existing baselines, as demonstrated through extensive experiments on curated benchmarks.
Generating interaction-centric videos, such as those depicting humans or robots interacting with objects, is crucial for embodied intelligence, as they provide rich and diverse visual priors for robot learning, manipulation policy training, and affordance reasoning. However, existing methods often struggle to model such complex and dynamic interactions. While recent studies show that masks can serve as effective control signals and enhance generation quality, obtaining dense and precise mask annotations remains a major challenge for real-world use. To overcome this limitation, we introduce Mask2IV, a novel framework specifically designed for interaction-centric video generation. It adopts a decoupled two-stage pipeline that first predicts plausible motion trajectories for both actor and object, then generates a video conditioned on these trajectories. This design eliminates the need for dense mask inputs from users while preserving the flexibility to manipulate the interaction process. Furthermore, Mask2IV supports versatile and intuitive control, allowing users to specify the target object of interaction and guide the motion trajectory through action descriptions or spatial position cues. To support systematic training and evaluation, we curate two benchmarks covering diverse action and object categories across both human-object interaction and robotic manipulation scenarios. Extensive experiments demonstrate that our method achieves superior visual realism and controllability compared to existing baselines.