Align-then-Slide: A complete evaluation framework for Ultra-Long Document-Level Machine Translation
This addresses the challenge of accurately evaluating doc-MT systems for researchers and practitioners, though it is incremental as it builds on existing evaluation methods by adapting them to whole-document outputs.
The paper tackles the problem of evaluating ultra-long document-level machine translation (doc-MT) by introducing Align-then-Slide, a framework that automatically aligns source and target sentences and uses sliding chunks for multi-granularity assessment, achieving a Pearson correlation of 0.929 with expert rankings on the WMT benchmark and enabling effective training methods like CPO and GRPO.
Large language models (LLMs) have ushered in a new era for document-level machine translation (\textit{doc}-mt), yet their whole-document outputs challenge existing evaluation methods that assume sentence-by-sentence alignment. We introduce \textit{\textbf{Align-then-Slide}}, a complete evaluation framework for ultra-long doc-mt. In the Align stage, we automatically infer sentence-level source-target correspondences and rebuild the target to match the source sentence number, resolving omissions and many-to-one/one-to-many mappings. In the n-Chunk Sliding Evaluate stage, we calculate averaged metric scores under 1-, 2-, 3- and 4-chunk for multi-granularity assessment. Experiments on the WMT benchmark show a Pearson correlation of 0.929 between our method with expert MQM rankings. On a newly curated real-world test set, our method again aligns closely with human judgments. Furthermore, preference data produced by Align-then-Slide enables effective CPO training and its direct use as a reward model for GRPO, both yielding translations preferred over a vanilla SFT baseline. The results validate our framework as an accurate, robust, and actionable evaluation tool for doc-mt systems.