Hear What Matters! Text-conditioned Selective Video-to-Audio Generation
This addresses a crucial need in multimedia production for precise audio editing and mixing by enabling selective sound generation from videos, though it is incremental as it builds on existing video-to-audio methods.
The paper tackles the problem of generating only user-intended sounds from multi-object videos by introducing a text-conditioned selective video-to-audio generation task, and proposes SelVA, which achieves effectiveness in audio quality, semantic alignment, and temporal synchronization as verified on the VGG-MONOAUDIO benchmark.
This work introduces a new task, text-conditioned selective video-to-audio (V2A) generation, which produces only the user-intended sound from a multi-object video. This capability is especially crucial in multimedia production, where audio tracks are handled individually for each sound source for precise editing, mixing, and creative control. However, current approaches generate single source-mixed sounds at once, largely because visual features are entangled, and region cues or prompts often fail to specify the source. We propose SelVA, a novel text-conditioned V2A model that treats the text prompt as an explicit selector of target source and modulates video encoder to distinctly extract prompt-relevant video features. The proposed supplementary tokens promote cross-attention by suppressing text-irrelevant activations with efficient parameter tuning, yielding robust semantic and temporal grounding. SelVA further employs a self-augmentation scheme to overcome the lack of mono audio track supervision. We evaluate SelVA on VGG-MONOAUDIO, a curated benchmark of clean single-source videos for such a task. Extensive experiments and ablations consistently verify its effectiveness across audio quality, semantic alignment, and temporal synchronization. Code and demo are available at https://jnwnlee.github.io/selva-demo/.