Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation
This enables sparse motion control in multi-object video generation, which is an incremental advancement for applications in video synthesis and editing.
The paper tackles the problem of generating videos from a single frame and sparse motion input, achieving realistic object-to-object interactions without explicit segmentation or motion data during training, with results showing it is on par or better than state-of-the-art methods in controllability and video quality.
We propose a novel unsupervised method to autoregressively generate videos from a single frame and a sparse motion input. Our trained model can generate unseen realistic object-to-object interactions. Although our model has never been given the explicit segmentation and motion of each object in the scene during training, it is able to implicitly separate their dynamics and extents. Key components in our method are the randomized conditioning scheme, the encoding of the input motion control, and the randomized and sparse sampling to enable generalization to out of distribution but realistic correlations. Our model, which we call YODA, has therefore the ability to move objects without physically touching them. Through extensive qualitative and quantitative evaluations on several datasets, we show that YODA is on par with or better than state of the art video generation prior work in terms of both controllability and video quality.