Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits
This addresses the challenge of maintaining audio-visual consistency in multimedia editing for content creators, though it appears incremental as it builds on existing video editing techniques.
The paper tackles the problem of joint audio-visual editing by enhancing coherence between edited video and audio, achieving improved audio-visual alignment and content integrity compared to existing approaches.
We introduce a novel pipeline for joint audio-visual editing that enhances the coherence between edited video and its accompanying audio. Our approach first applies state-of-the-art video editing techniques to produce the target video, then performs audio editing to align with the visual changes. To achieve this, we present a new video-to-audio generation model that conditions on the source audio, target video, and a text prompt. We extend the model architecture to incorporate conditional audio input and propose a data augmentation strategy that improves training efficiency. Furthermore, our model dynamically adjusts the influence of the source audio based on the complexity of the edits, preserving the original audio structure where possible. Experimental results demonstrate that our method outperforms existing approaches in maintaining audio-visual alignment and content integrity.