COMET-poly: Machine Translation Metric Grounded in Other Candidates
This addresses a discrepancy in evaluation setup for machine translation metrics, offering incremental improvements for researchers and practitioners in NLP.
The paper tackles the problem that automated machine translation metrics typically evaluate single translations without context, unlike humans who compare multiple alternatives, by proposing COMET-polycand and COMET-polyic metrics that incorporate additional translations or retrieved examples, resulting in improved segment-level performance with Kendall's tau-b correlations increasing from 0.079 to 0.118 and 0.116, respectively.
Automated metrics for machine translation attempt to replicate human judgment. Unlike humans, who often assess a translation in the context of multiple alternatives, these metrics typically consider only the source sentence and a single translation. This discrepancy in the evaluation setup may negatively impact the performance of automated metrics. We propose two automated metrics that incorporate additional information beyond the single translation. COMET-polycand uses alternative translations of the same source sentence to compare and contrast with the translation at hand, thereby providing a more informed assessment of its quality. COMET-polyic, inspired by retrieval-based in-context learning, takes in translations of similar source texts along with their human-labeled quality scores to guide the evaluation. We find that including a single additional translation in COMET-polycand improves the segment-level metric performance (0.079 to 0.118 Kendall's tau-b correlation), with further gains when more translations are added. Incorporating retrieved examples in COMET-polyic yields similar improvements (0.079 to 0.116 Kendall's tau-b correlation). We release our models publicly.