Automatic Song Translation for Tonal Languages
This addresses the problem of translating songs for tonal languages like Mandarin, which is incremental as it builds on existing translation methods by adding tonal and melodic constraints.
The paper tackles automatic song translation for tonal languages by addressing the challenge of aligning word tones with melody while preserving meaning, proposing criteria for meaning, singability, and intelligibility, and developing an unsupervised system called GagaST that combines pre-training with decoding constraints, which shows successful balancing of semantics and singability in evaluations.
This paper develops automatic song translation (AST) for tonal languages and addresses the unique challenge of aligning words' tones with melody of a song in addition to conveying the original meaning. We propose three criteria for effective AST -- preserving meaning, singability and intelligibility -- and design metrics for these criteria. We develop a new benchmark for English--Mandarin song translation and develop an unsupervised AST system, Guided AliGnment for Automatic Song Translation (GagaST), which combines pre-training with three decoding constraints. Both automatic and human evaluations show GagaST successfully balances semantics and singability.