VideoAR: Autoregressive Video Generation via Next-Frame & Scale Prediction
This work addresses the challenge of scalable and efficient video generation for AI and multimedia applications, representing an incremental advancement by narrowing the performance gap between autoregressive and diffusion models.
The paper tackles the problem of computationally intensive video generation by introducing VideoAR, a large-scale visual autoregressive framework that combines multi-scale next-frame prediction with autoregressive modeling, achieving state-of-the-art results among autoregressive models with improvements such as reducing FVD on UCF-101 from 99.5 to 88.6 and cutting inference steps by over 10x.
Recent advances in video generation have been dominated by diffusion and flow-matching models, which produce high-quality results but remain computationally intensive and difficult to scale. In this work, we introduce VideoAR, the first large-scale Visual Autoregressive (VAR) framework for video generation that combines multi-scale next-frame prediction with autoregressive modeling. VideoAR disentangles spatial and temporal dependencies by integrating intra-frame VAR modeling with causal next-frame prediction, supported by a 3D multi-scale tokenizer that efficiently encodes spatio-temporal dynamics. To improve long-term consistency, we propose Multi-scale Temporal RoPE, Cross-Frame Error Correction, and Random Frame Mask, which collectively mitigate error propagation and stabilize temporal coherence. Our multi-stage pretraining pipeline progressively aligns spatial and temporal learning across increasing resolutions and durations. Empirically, VideoAR achieves new state-of-the-art results among autoregressive models, improving FVD on UCF-101 from 99.5 to 88.6 while reducing inference steps by over 10x, and reaching a VBench score of 81.74-competitive with diffusion-based models an order of magnitude larger. These results demonstrate that VideoAR narrows the performance gap between autoregressive and diffusion paradigms, offering a scalable, efficient, and temporally consistent foundation for future video generation research.