MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing
This work addresses the challenge of enhancing movie dubbing quality for applications in entertainment and media, though it appears incremental as it builds on existing multi-modal and speech generation technologies.
The paper tackles the problem of insufficient exploration in movie dubbing for aspects like dubbing styles, dialogue handling, and subtle details such as speaker age and gender, by introducing a multi-modal generative framework that uses vision-language models and speech generation, resulting in improvements of up to 1.09% to 19.08% on benchmark metrics compared to state-of-the-art methods.
Current movie dubbing technology can produce the desired speech using a reference voice and input video, maintaining perfect synchronization with the visuals while effectively conveying the intended emotions. However, crucial aspects of movie dubbing, including adaptation to various dubbing styles, effective handling of dialogue, narration, and monologues, as well as consideration of subtle details such as speaker age and gender, remain insufficiently explored. To tackle these challenges, we introduce a multi-modal generative framework. First, it utilizes a multi-modal large vision-language model (VLM) to analyze visual inputs, enabling the recognition of dubbing types and fine-grained attributes. Second, it produces high-quality dubbing using large speech generation models, guided by multi-modal inputs. Additionally, a movie dubbing dataset with annotations for dubbing types and subtle details is constructed to enhance movie understanding and improve dubbing quality for the proposed multi-modal framework. Experimental results across multiple benchmark datasets show superior performance compared to state-of-the-art (SOTA) methods. In details, the LSE-D, SPK-SIM, EMO-SIM, and MCD exhibit improvements of up to 1.09%, 8.80%, 19.08%, and 18.74%, respectively.