FlashVideo: A Framework for Swift Inference in Text-to-Video Generation
This addresses the bottleneck of prolonged inference for generating short video clips like GIFs, offering a domain-specific advancement in video generation.
The paper tackles the problem of slow inference times in text-to-video generation by introducing FlashVideo, a framework that adapts the RetNet architecture to reduce time complexity from O(L^2) to O(L), achieving a 9.17x efficiency improvement over traditional autoregressive transformers.
In the evolving field of machine learning, video generation has witnessed significant advancements with autoregressive-based transformer models and diffusion models, known for synthesizing dynamic and realistic scenes. However, these models often face challenges with prolonged inference times, even for generating short video clips such as GIFs. This paper introduces FlashVideo, a novel framework tailored for swift Text-to-Video generation. FlashVideo represents the first successful adaptation of the RetNet architecture for video generation, bringing a unique approach to the field. Leveraging the RetNet-based architecture, FlashVideo reduces the time complexity of inference from $\mathcal{O}(L^2)$ to $\mathcal{O}(L)$ for a sequence of length $L$, significantly accelerating inference speed. Additionally, we adopt a redundant-free frame interpolation method, enhancing the efficiency of frame interpolation. Our comprehensive experiments demonstrate that FlashVideo achieves a $\times9.17$ efficiency improvement over a traditional autoregressive-based transformer model, and its inference speed is of the same order of magnitude as that of BERT-based transformer models.