Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity
This work addresses efficiency challenges in video generation for AI researchers and practitioners, though it appears incremental as it builds on existing autoregressive frameworks.
The paper tackles the problem of high computational costs and prolonged training times in autoregressive video generation models by proposing a Local Optimization method with Representation Continuity strategy, which reduces training cost by half while maintaining or improving video quality.
Autoregressive models have shown superior performance and efficiency in image generation, but remain constrained by high computational costs and prolonged training times in video generation. In this study, we explore methods to accelerate training for autoregressive video generation models through empirical analyses. Our results reveal that while training on fewer video frames significantly reduces training time, it also exacerbates error accumulation and introduces inconsistencies in the generated videos. To address these issues, we propose a Local Optimization (Local Opt.) method, which optimizes tokens within localized windows while leveraging contextual information to reduce error propagation. Inspired by Lipschitz continuity, we propose a Representation Continuity (ReCo) strategy to improve the consistency of generated videos. ReCo utilizes continuity loss to constrain representation changes, improving model robustness and reducing error accumulation. Extensive experiments on class- and text-to-video datasets demonstrate that our approach achieves superior performance to the baseline while halving the training cost without sacrificing quality.