DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos
This addresses the challenge of precise audio-visual synchronization in video generation, which is important for applications like multimedia and entertainment, though it appears incremental as it builds on existing multi-modal frameworks.
The paper tackles the problem of poor alignment between visual and generated audio domains in video-to-audio synthesis by proposing a framework that uses a multi-modal large language model's chain-of-thought for step-by-step reasoning without extra annotations, achieving competitive performance with metrics showing reductions in misalignment indicators by up to 38.61% and improvements in quality indicators by up to 6.39%.
Currently, high-quality, synchronized audio is synthesized from video and optional text inputs using various multi-modal joint learning frameworks. However, the precise alignment between the visual and generated audio domains remains far from satisfactory. One key factor is the lack of sufficient temporal and semantic alignment annotations in open-source video-audio and text-audio benchmarks. Therefore, we propose a framework for audio generation from videos, leveraging the internal chain-of-thought (CoT) of a multi-modal large language model (MLLM) to enable step-by-step reasoning without requiring additional annotations. Additionally, a corresponding multi-modal reasoning dataset is constructed to facilitate the learning of initial reasoning in audio generation. In the experiments, we demonstrate the effectiveness of the proposed framework in reducing misalignment (voice-over) in generated audio and achieving competitive performance compared to various state-of-the-art models. The evaluation results show that the proposed method outperforms state-of-the-art approaches across multiple metrics. Specifically, the F DP aSST indicator is reduced by up to 10.07%, the F DP AN N s indicator by up to 11.62%, and the F DV GG indicator by up to 38.61%. Furthermore, the IS indicator improves by up to 4.95%, the IB-score indicator increases by up to 6.39%, and the DeSync indicator is reduced by up to 0.89%.