Conformalizing Machine Translation Evaluation
This work addresses the need for reliable uncertainty estimation in machine translation evaluation, which is crucial for practitioners to trust model predictions, though it is incremental as it builds on existing conformal prediction techniques.
The paper tackled the problem of unreliable uncertainty estimation in machine translation evaluation by showing that existing methods often underestimate uncertainty and produce misleading confidence intervals. It proposed using conformal prediction to correct these intervals, achieving theoretically guaranteed coverage and addressing biases across language pairs and translation quality.
Several uncertainty estimation methods have been recently proposed for machine translation evaluation. While these methods can provide a useful indication of when not to trust model predictions, we show in this paper that the majority of them tend to underestimate model uncertainty, and as a result they often produce misleading confidence intervals that do not cover the ground truth. We propose as an alternative the use of conformal prediction, a distribution-free method to obtain confidence intervals with a theoretically established guarantee on coverage. First, we demonstrate that split conformal prediction can ``correct'' the confidence intervals of previous methods to yield a desired coverage level. Then, we highlight biases in estimated confidence intervals, both in terms of the translation language pairs and the quality of translations. We apply conditional conformal prediction techniques to obtain calibration subsets for each data subgroup, leading to equalized coverage.