Motion Prior Distillation in Time Reversal Sampling for Generative Inbetweening
This work addresses a specific bottleneck in video generation for applications like animation or video editing, representing an incremental improvement over existing inference-time sampling methods.
The paper tackled the problem of temporal discontinuities and visual artifacts in generative inbetweening with image-to-video diffusion models by proposing Motion Prior Distillation, which distills motion residuals to align paths, resulting in more temporally coherent outputs as validated by quantitative benchmarks and user studies.
Recent progress in image-to-video (I2V) diffusion models has significantly advanced the field of generative inbetweening, which aims to generate semantically plausible frames between two keyframes. In particular, inference-time sampling strategies, which leverage the generative priors of large-scale pre-trained I2V models without additional training, have become increasingly popular. However, existing inference-time sampling, either fusing forward and backward paths in parallel or alternating them sequentially, often suffers from temporal discontinuities and undesirable visual artifacts due to the misalignment between the two generated paths. This is because each path follows the motion prior induced by its own conditioning frame. In this work, we propose Motion Prior Distillation (MPD), a simple yet effective inference-time distillation technique that suppresses bidirectional mismatch by distilling the motion residual of the forward path into the backward path. Our method can deliberately avoid denoising the end-conditioned path which causes the ambiguity of the path, and yield more temporally coherent inbetweening results with the forward motion prior. We not only perform quantitative evaluations on standard benchmarks, but also conduct extensive user studies to demonstrate the effectiveness of our approach in practical scenarios.