Language translation, and change of accent for speech-to-speech task using diffusion model
This addresses the need for cross-cultural communication by handling both language translation and accent adaptation in speech-to-speech tasks, representing an incremental advancement in the field.
The paper tackles the problem of simultaneous speech translation and accent adaptation, proposing a unified diffusion model approach that achieves joint optimization, resulting in a more parameter-efficient and effective model compared to traditional pipelines.
Speech-to-speech translation (S2ST) aims to convert spoken input in one language to spoken output in another, typically focusing on either language translation or accent adaptation. However, effective cross-cultural communication requires handling both aspects simultaneously - translating content while adapting the speaker's accent to match the target language context. In this work, we propose a unified approach for simultaneous speech translation and change of accent, a task that remains underexplored in current literature. Our method reformulates the problem as a conditional generation task, where target speech is generated based on phonemes and guided by target speech features. Leveraging the power of diffusion models, known for high-fidelity generative capabilities, we adapt text-to-image diffusion strategies by conditioning on source speech transcriptions and generating Mel spectrograms representing the target speech with desired linguistic and accentual attributes. This integrated framework enables joint optimization of translation and accent adaptation, offering a more parameter-efficient and effective model compared to traditional pipelines.