Sequence-Adaptive Video Prediction in Continuous Streams using Diffusion Noise Optimization
This addresses the challenge of efficient adaptation for video prediction in streaming contexts, though it is incremental as it builds on existing diffusion models.
The paper tackles the problem of adapting diffusion-based video prediction models to continuous video streams by refining diffusion noise during inference instead of fine-tuning parameters, resulting in improved performance on metrics like FVD, SSIM, and PSNR across datasets such as Ego4D and OpenDV-YouTube.
In this work, we investigate diffusion-based video prediction models, which forecast future video frames, for continuous video streams. In this context, the models observe continuously new training samples, and we aim to leverage this to improve their predictions. We thus propose an approach that continuously adapts a pre-trained diffusion model to a video stream. Since fine-tuning the parameters of a large diffusion model is too expensive, we refine the diffusion noise during inference while keeping the model parameters frozen, allowing the model to adaptively determine suitable sampling noise. We term the approach Sequence Adaptive Video Prediction with Diffusion Noise Optimization (SAVi-DNO). To validate our approach, we introduce a new evaluation setting on the Ego4D dataset, focusing on simultaneous adaptation and evaluation on long continuous videos. Empirical results demonstrate improved performance based on FVD, SSIM, and PSNR metrics on long videos of Ego4D and OpenDV-YouTube, as well as videos of UCF-101 and SkyTimelapse, showcasing SAVi-DNO's effectiveness.