Average Token Delay: A Duration-aware Latency Metric for Simultaneous Translation
This addresses latency evaluation for users of simultaneous translation systems, but it is incremental as it refines an existing metric rather than introducing a new method.
The paper tackled the problem of evaluating latency in simultaneous translation, where existing metrics ignore translation duration, and proposed Average Token Delay (ATD) as a duration-aware metric, showing it had the highest correlation with Ear-Voice Span under most conditions.
Simultaneous translation is a task in which the translation begins before the end of an input speech segment. Its evaluation should be conducted based on latency in addition to quality, and for users, the smallest possible amount of latency is preferable. Most existing metrics measure latency based on the start timings of partial translations and ignore their duration. This means such metrics do not penalize the latency caused by long translation output, which delays the comprehension of users and subsequent translations. In this work, we propose a novel latency evaluation metric for simultaneous translation called \emph{Average Token Delay} (ATD) that focuses on the duration of partial translations. We demonstrate its effectiveness through analyses simulating user-side latency based on Ear-Voice Span (EVS). In our experiment, ATD had the highest correlation with EVS among baseline latency metrics under most conditions.