SALTED: A Framework for SAlient Long-Tail Translation Error Detection
This addresses the reliability issue in state-of-the-art MT systems for developers and users by offering a tool to detect previously invisible errors, though it is incremental as it builds on existing behavioral testing approaches.
The authors tackled the problem of detecting rare, unpredictable errors in neural machine translation (NMT) systems, such as number translation or hallucinations, by introducing SALTED, a framework that provides fine-grained visibility into these long-tail errors, enabling error identification, data filtering, and model fine-tuning.
Traditional machine translation (MT) metrics provide an average measure of translation quality that is insensitive to the long tail of behavioral problems in MT. Examples include translation of numbers, physical units, dropped content and hallucinations. These errors, which occur rarely and unpredictably in Neural Machine Translation (NMT), greatly undermine the reliability of state-of-the-art MT systems. Consequently, it is important to have visibility into these problems during model development. Towards this direction, we introduce SALTED, a specifications-based framework for behavioral testing of MT models that provides fine-grained views of salient long-tail errors, permitting trustworthy visibility into previously invisible problems. At the core of our approach is the development of high-precision detectors that flag errors (or alternatively, verify output correctness) between a source sentence and a system output. We demonstrate that such detectors could be used not just to identify salient long-tail errors in MT systems, but also for higher-recall filtering of the training data, fixing targeted errors with model fine-tuning in NMT and generating novel data for metamorphic testing to elicit further bugs in models.