IsoChronoMeter: A simple and effective isochronic translation evaluation metric
This addresses the need for scalable evaluation in automatic dubbing, but is incremental as it builds on existing TTS duration predictors.
The paper tackles the problem of evaluating isochronic translation for video dubbing by introducing IsoChronoMeter (ICM), a metric that measures translation isochrony without gold data, and uses it to show shortcomings in state-of-the-art translation systems.
Machine translation (MT) has come a long way and is readily employed in production systems to serve millions of users daily. With the recent advances in generative AI, a new form of translation is becoming possible - video dubbing. This work motivates the importance of isochronic translation, especially in the context of automatic dubbing, and introduces `IsoChronoMeter' (ICM). ICM is a simple yet effective metric to measure isochrony of translations in a scalable and resource-efficient way without the need for gold data, based on state-of-the-art text-to-speech (TTS) duration predictors. We motivate IsoChronoMeter and demonstrate its effectiveness. Using ICM we demonstrate the shortcomings of state-of-the-art translation systems and show the need for new methods. We release the code at this URL: \url{https://github.com/braskai/isochronometer}.