Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages
This work addresses the problem of unreliable evaluation for Indian languages, spoken by over 1.5 billion people, by providing a systematic benchmark, though it is incremental as it extends existing metric evaluation practices to new languages.
The paper tackled the lack of evaluation metrics for Indian languages in machine translation and summarization by introducing ITEM, a benchmark that assessed 26 automatic metrics across six languages, finding that LLM-based evaluators aligned best with human judgments and highlighting issues like outlier sensitivity and metric robustness.
While automatic metrics drive progress in Machine Translation (MT) and Text Summarization (TS), existing metrics have been developed and validated almost exclusively for English and other high-resource languages. This narrow focus leaves Indian languages, spoken by over 1.5 billion people, largely overlooked, casting doubt on the universality of current evaluation practices. To address this gap, we introduce ITEM, a large-scale benchmark that systematically evaluates the alignment of 26 automatic metrics with human judgments across six major Indian languages, enriched with fine-grained annotations. Our extensive evaluation, covering agreement with human judgments, sensitivity to outliers, language-specific reliability, inter-metric correlations, and resilience to controlled perturbations, reveals four central findings: (1) LLM-based evaluators show the strongest alignment with human judgments at both segment and system levels; (2) outliers exert a significant impact on metric-human agreement; (3) in TS, metrics are more effective at capturing content fidelity, whereas in MT, they better reflect fluency; and (4) metrics differ in their robustness and sensitivity when subjected to diverse perturbations. Collectively, these findings offer critical guidance for advancing metric design and evaluation in Indian languages.