Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers
This addresses the issue of model reliability in machine translation for users by providing a method to detect inputs outside the training distribution, though it is incremental as it builds on existing Bayesian deep learning and Transformer approaches.
The paper tackled the problem of detecting out-of-distribution sentences in neural machine translation by developing a new uncertainty measure for long sequences of discrete variables, and showed that it successfully identified Dutch sentences as out-of-distribution when the model was trained on German-English translation using WMT13 and Europarl datasets.
We detect out-of-training-distribution sentences in Neural Machine Translation using the Bayesian Deep Learning equivalent of Transformer models. For this we develop a new measure of uncertainty designed specifically for long sequences of discrete random variables -- i.e. words in the output sentence. Our new measure of uncertainty solves a major intractability in the naive application of existing approaches on long sentences. We use our new measure on a Transformer model trained with dropout approximate inference. On the task of German-English translation using WMT13 and Europarl, we show that with dropout uncertainty our measure is able to identify when Dutch source sentences, sentences which use the same word types as German, are given to the model instead of German.