Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Framework
This work addresses theoretical gaps in video generation models for researchers and practitioners, offering incremental improvements through a unified analysis and mitigation strategies.
The paper tackles the lack of theoretical analysis for Auto-Regressive Video Diffusion Models (ARVDMs) by developing a unified framework called Meta-ARVDM, which reveals error accumulation and an unavoidable memory bottleneck, and proposes methods to mitigate these issues, achieving improved performance validated on DMLab and Minecraft datasets.
A variety of Auto-Regressive Video Diffusion Models (ARVDM) have achieved remarkable successes in generating realistic long-form videos. However, theoretical analyses of these models remain scant. In this work, we develop theoretical underpinnings for these models and use our insights to improve the performance of existing models. We first develop Meta-ARVDM, a unified framework of ARVDMs that subsumes most existing methods. Using Meta-ARVDM, we analyze the KL-divergence between the videos generated by Meta-ARVDM and the true videos. Our analysis uncovers two important phenomena inherent to ARVDM -- error accumulation and memory bottleneck. By deriving an information-theoretic impossibility result, we show that the memory bottleneck phenomenon cannot be avoided. To mitigate the memory bottleneck, we design various network structures to explicitly use more past frames. We also achieve a significantly improved trade-off between the mitigation of the memory bottleneck and the inference efficiency by compressing the frames. Experimental results on DMLab and Minecraft validate the efficacy of our methods. Our experiments also demonstrate a Pareto-frontier between the error accumulation and memory bottleneck across different methods.