Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space
This work addresses the high computational cost and loss of temporal cues in video-to-audio generation for researchers and practitioners, offering an efficient single-stage solution with improved synchronization.
Flowley achieves state-of-the-art video-to-audio generation on VGGSound using a single-stage end-to-end architecture with Progressive Soft-masked Cross-Attention, outperforming existing methods without pretrained alignment modules and further exceeding closed-source methods in zero-shot audio quality when combined with SoundCap.
Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.