CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translation
This work addresses a critical evaluation issue for researchers and developers in simultaneous speech translation, though it is incremental as it modifies existing metrics rather than introducing a new paradigm.
The paper tackled the problem of unrealistic latency measurements in simultaneous speech translation systems by identifying a fundamental misconception in existing evaluation approaches, and proposed a modification to correctly measure computation-aware latency, demonstrating its impact across different metrics.
Simultaneous speech translation (SimulST) systems must balance translation quality with response time, making latency measurement crucial for evaluating their real-world performance. However, there has been a longstanding belief that current metrics yield unrealistically high latency measurements in unsegmented streaming settings. In this paper, we investigate this phenomenon, revealing its root cause in a fundamental misconception underlying existing latency evaluation approaches. We demonstrate that this issue affects not only streaming but also segment-level latency evaluation across different metrics. Furthermore, we propose a modification to correctly measure computation-aware latency for SimulST systems, addressing the limitations present in existing metrics.