Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
This addresses the challenge of realistic video-to-audio synthesis for applications in multimedia and entertainment, representing a strong specific gain rather than a foundational breakthrough.
The paper tackles the problem of generating high-quality audio synchronized with video content by proposing Kling-Foley, a multimodal diffusion transformer model that achieves new state-of-the-art performance in audio-visual generation across metrics like distribution matching and alignment.
We propose Kling-Foley, a large-scale multimodal Video-to-Audio generation model that synthesizes high-quality audio synchronized with video content. In Kling-Foley, we introduce multimodal diffusion transformers to model the interactions between video, audio, and text modalities, and combine it with a visual semantic representation module and an audio-visual synchronization module to enhance alignment capabilities. Specifically, these modules align video conditions with latent audio elements at the frame level, thereby improving semantic alignment and audio-visual synchronization. Together with text conditions, this integrated approach enables precise generation of video-matching sound effects. In addition, we propose a universal latent audio codec that can achieve high-quality modeling in various scenarios such as sound effects, speech, singing, and music. We employ a stereo rendering method that imbues synthesized audio with a spatial presence. At the same time, in order to make up for the incomplete types and annotations of the open-source benchmark, we also open-source an industrial-level benchmark Kling-Audio-Eval. Our experiments show that Kling-Foley trained with the flow matching objective achieves new audio-visual SOTA performance among public models in terms of distribution matching, semantic alignment, temporal alignment and audio quality.