Moaw: Unleashing Motion Awareness for Video Diffusion Models
This work addresses the need for more unified and controllable video learning frameworks by bridging generative modeling and motion understanding, though it appears incremental as it builds on existing capabilities of video diffusion models.
The authors tackled the problem of enhancing motion awareness in video diffusion models by proposing Moaw, a framework that shifts a diffusion model from image-to-video generation to video-to-dense-tracking, enabling motion transfer without additional adapters and achieving zero-shot adaptation due to network homogeneity.
Video diffusion models, trained on large-scale datasets, naturally capture correspondences of shared features across frames. Recent works have exploited this property for tasks such as optical flow prediction and tracking in a zero-shot setting. Motivated by these findings, we investigate whether supervised training can more fully harness the tracking capability of video diffusion models. To this end, we propose Moaw, a framework that unleashes motion awareness for video diffusion models and leverages it to facilitate motion transfer. Specifically, we train a diffusion model for motion perception, shifting its modality from image-to-video generation to video-to-dense-tracking. We then construct a motion-labeled dataset to identify features that encode the strongest motion information, and inject them into a structurally identical video generation model. Owing to the homogeneity between the two networks, these features can be naturally adapted in a zero-shot manner, enabling motion transfer without additional adapters. Our work provides a new paradigm for bridging generative modeling and motion understanding, paving the way for more unified and controllable video learning frameworks.