$^R$FLAV: Rolling Flow matching for infinite Audio Video generation
It solves the problem of generating synchronized and coherent audio-video content for applications in media and AI, representing an incremental advancement.
The paper tackles joint audio-video generation by addressing challenges in quality, synchronization, and infinite duration, achieving state-of-the-art performance in multimodal tasks.
Joint audio-video (AV) generation is still a significant challenge in generative AI, primarily due to three critical requirements: quality of the generated samples, seamless multimodal synchronization and temporal coherence, with audio tracks that match the visual data and vice versa, and limitless video duration. In this paper, we present $^R$-FLAV, a novel transformer-based architecture that addresses all the key challenges of AV generation. We explore three distinct cross modality interaction modules, with our lightweight temporal fusion module emerging as the most effective and computationally efficient approach for aligning audio and visual modalities. Our experimental results demonstrate that $^R$-FLAV outperforms existing state-of-the-art models in multimodal AV generation tasks. Our code and checkpoints are available at https://github.com/ErgastiAlex/R-FLAV.