Boximator: Generating Rich and Controllable Motions for Video Synthesis
This addresses the problem of precise motion control in video synthesis for users in fields like animation or content creation, representing an incremental improvement as a plug-in for existing models.
The paper tackles the challenge of generating rich and controllable motion in video synthesis by proposing Boximator, a plug-in approach that uses hard and soft box constraints for fine-grained control over object position, shape, or motion paths, achieving state-of-the-art video quality scores (FVD) and improved bounding box alignment.
Generating rich and controllable motion is a pivotal challenge in video synthesis. We propose Boximator, a new approach for fine-grained motion control. Boximator introduces two constraint types: hard box and soft box. Users select objects in the conditional frame using hard boxes and then use either type of boxes to roughly or rigorously define the object's position, shape, or motion path in future frames. Boximator functions as a plug-in for existing video diffusion models. Its training process preserves the base model's knowledge by freezing the original weights and training only the control module. To address training challenges, we introduce a novel self-tracking technique that greatly simplifies the learning of box-object correlations. Empirically, Boximator achieves state-of-the-art video quality (FVD) scores, improving on two base models, and further enhanced after incorporating box constraints. Its robust motion controllability is validated by drastic increases in the bounding box alignment metric. Human evaluation also shows that users favor Boximator generation results over the base model.