On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
This addresses robustness challenges in neural machine translation for practitioners dealing with domain shifts, though it is incremental as it builds on existing methods.
The paper tackles the problem of exposure bias in neural machine translation, linking it to hallucinations under domain shift, and shows that Minimum Risk Training can mitigate this issue, improving model robustness across multiple test domains.
The standard training algorithm in neural machine translation (NMT) suffers from exposure bias, and alternative algorithms have been proposed to mitigate this. However, the practical impact of exposure bias is under debate. In this paper, we link exposure bias to another well-known problem in NMT, namely the tendency to generate hallucinations under domain shift. In experiments on three datasets with multiple test domains, we show that exposure bias is partially to blame for hallucinations, and that training with Minimum Risk Training, which avoids exposure bias, can mitigate this. Our analysis explains why exposure bias is more problematic under domain shift, and also links exposure bias to the beam search problem, i.e. performance deterioration with increasing beam size. Our results provide a new justification for methods that reduce exposure bias: even if they do not increase performance on in-domain test sets, they can increase model robustness to domain shift.