Foley-Omni: A Unified Multimodal Generation Model from Task-Level Audio Synthesis to Complete Video Soundtrack Generation
For video production, Foley-Omni enables joint and consistent generation of multiple audio components from a single model, addressing a practical bottleneck in real-world video soundtrack creation.
Foley-Omni extends isolated task-level audio synthesis to complete video soundtrack generation by jointly modeling speech, sound effects, and music. It achieves competitive performance with expert systems on individual tasks while improving speech intelligibility, audiovisual consistency, and perceptual quality for mixed soundtrack generation.
Recent unified audio generation models can support diverse tasks across speech, sound effects, and music, but most of them still focus on isolated task-level synthesis. However, real video production often requires multiple components of a complete audio track to be generated jointly and consistently for the same video. We present Foley-Omni, a unified multimodal audio generation model that extends isolated task-level synthesis to complete video soundtrack generation by jointly modeling speech, sound effects, and music within a shared latent generation process. To support training and reproducible evaluation, we develop an audiovisual data curation pipeline and introduce V2ST-Bench, a benchmark for holistic video soundtrack generation evaluation. Experiments show that Foley-Omni achieves competitive performance with expert systems on individual synthesis tasks, while improving speech intelligibility, audiovisual consistency and perceptual quality for mixed soundtrack generation.