Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models
This addresses the challenge of reliable long-form translation evaluation for NLP researchers and practitioners, though it is incremental as it builds on existing LLM-based evaluation methods.
The study tackled the problem of evaluating long documents in machine translation using large language models (LLMs), finding that longer texts lead to fewer error spans and reduced ranking accuracy, but methods like Focus Sentence Prompting and fine-tuning largely mitigated this bias.
Accurately evaluating machine-translated text remains a long-standing challenge, particularly for long documents. Recent work has shown that large language models (LLMs) can serve as reliable and interpretable sentence-level translation evaluators via MQM error span annotations. With modern LLMs supporting larger context windows, a natural question arises: can we feed entire document translations into an LLM for quality assessment? Ideally, evaluation should be invariant to text length, producing consistent error spans regardless of input granularity. However, our analysis shows that text length significantly impacts evaluation: longer texts lead to fewer error spans and reduced system ranking accuracy. To address this limitation, we evaluate several strategies, including granularity-aligned prompting, Focus Sentence Prompting (FSP), and a fine-tuning approach to better align LLMs with the evaluation task. The latter two methods largely mitigate this length bias, making LLMs more reliable for long-form translation evaluation.