RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks
This addresses the challenge of computationally intensive and artifact-prone video generation for applications in media and simulation, representing a strong incremental advance with specific efficiency gains.
The paper tackles the problem of generating high-fidelity, long-duration videos efficiently by proposing a novel unconditional video generative model that uses a hybrid tri-plane representation and a single latent code, reducing computational complexity by more than half compared to state-of-the-art methods and enabling synthesis of 256x256 pixel videos over 5 seconds at 30 fps.
We present a novel unconditional video generative model designed to address long-term spatial and temporal dependencies, with attention to computational and dataset efficiency. To capture long spatio-temporal dependencies, our approach incorporates a hybrid explicit-implicit tri-plane representation inspired by 3D-aware generative frameworks developed for three-dimensional object representation and employs a single latent code to model an entire video clip. Individual video frames are then synthesized from an intermediate tri-plane representation, which itself is derived from the primary latent code. This novel strategy more than halves the computational complexity measured in FLOPs compared to the most efficient state-of-the-art methods. Consequently, our approach facilitates the efficient and temporally coherent generation of videos. Moreover, our joint frame modeling approach, in contrast to autoregressive methods, mitigates the generation of visual artifacts. We further enhance the model's capabilities by integrating an optical flow-based module within our Generative Adversarial Network (GAN) based generator architecture, thereby compensating for the constraints imposed by a smaller generator size. As a result, our model synthesizes high-fidelity video clips at a resolution of $256\times256$ pixels, with durations extending to more than $5$ seconds at a frame rate of 30 fps. The efficacy and versatility of our approach are empirically validated through qualitative and quantitative assessments across three different datasets comprising both synthetic and real video clips. We will make our training and inference code public.