MoAlign: Motion-Centric Representation Alignment for Video Diffusion Models
This addresses the issue of poor motion quality in video generation for applications like content creation, but it is incremental as it builds on existing alignment methods by focusing on disentanglement.
The paper tackled the problem of text-to-video diffusion models generating temporally incoherent and physically implausible motion by proposing a motion-centric alignment framework that learns a disentangled motion subspace from a pretrained video encoder, resulting in improved physical commonsense in a state-of-the-art model as evidenced by evaluations on multiple benchmarks and a user study.
Text-to-video diffusion models have enabled high-quality video synthesis, yet often fail to generate temporally coherent and physically plausible motion. A key reason is the models' insufficient understanding of complex motions that natural videos often entail. Recent works tackle this problem by aligning diffusion model features with those from pretrained video encoders. However, these encoders mix video appearance and dynamics into entangled features, limiting the benefit of such alignment. In this paper, we propose a motion-centric alignment framework that learns a disentangled motion subspace from a pretrained video encoder. This subspace is optimized to predict ground-truth optical flow, ensuring it captures true motion dynamics. We then align the latent features of a text-to-video diffusion model to this new subspace, enabling the generative model to internalize motion knowledge and generate more plausible videos. Our method improves the physical commonsense in a state-of-the-art video diffusion model, while preserving adherence to textual prompts, as evidenced by empirical evaluations on VideoPhy, VideoPhy2, VBench, and VBench-2.0, along with a user study.