Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better
This addresses hallucinations in machine translation, a critical issue for translation quality, but it is incremental as it builds on existing detection methods.
The paper tackles the problem of hallucinations in neural machine translation by proposing a method that uses the translation model's internal source contribution to detect and mitigate severe hallucinations, improving detection accuracy by a factor of 2 and matching previous best external-model approaches. It also shows that using external sentence similarity from cross-lingual embeddings further enhances these results.
While the problem of hallucinations in neural machine translation has long been recognized, so far the progress on its alleviation is very little. Indeed, recently it turned out that without artificially encouraging models to hallucinate, previously existing methods fall short and even the standard sequence log-probability is more informative. It means that characteristics internal to the model can give much more information than we expect, and before using external models and measures, we first need to ask: how far can we go if we use nothing but the translation model itself ? We propose to use a method that evaluates the percentage of the source contribution to a generated translation. Intuitively, hallucinations are translations "detached" from the source, hence they can be identified by low source contribution. This method improves detection accuracy for the most severe hallucinations by a factor of 2 and is able to alleviate hallucinations at test time on par with the previous best approach that relies on external models. Next, if we move away from internal model characteristics and allow external tools, we show that using sentence similarity from cross-lingual embeddings further improves these results.