Efficient Continuous Video Flow Model for Video Prediction
This addresses latency bottlenecks for video prediction tasks, enabling faster generation of frames in applications like video synthesis, though it is incremental as it builds on existing multi-step methods.
The paper tackles the high latency of multi-step video prediction models like diffusion and rectified flow by proposing a novel approach that reduces sampling steps and model size to one-third, achieving state-of-the-art performance on benchmarks such as KTH, BAIR, Human3.6M, and UCF101.
Multi-step prediction models, such as diffusion and rectified flow models, have emerged as state-of-the-art solutions for generation tasks. However, these models exhibit higher latency in sampling new frames compared to single-step methods. This latency issue becomes a significant bottleneck when adapting such methods for video prediction tasks, given that a typical 60-second video comprises approximately 1.5K frames. In this paper, we propose a novel approach to modeling the multi-step process, aimed at alleviating latency constraints and facilitating the adaptation of such processes for video prediction tasks. Our approach not only reduces the number of sample steps required to predict the next frame but also minimizes computational demands by reducing the model size to one-third of the original size. We evaluate our method on standard video prediction datasets, including KTH, BAIR action robot, Human3.6M and UCF101, demonstrating its efficacy in achieving state-of-the-art performance on these benchmarks.