Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study
This addresses safety and ethical issues for users relying on machine translation in sensitive domains like mental health, though it is incremental in highlighting a known bottleneck.
The paper tackles the problem of fluent but critical errors in neural machine translation (NMT) systems, particularly for Arabic mental health tweets, by introducing a manually annotated dataset of such errors and showing that common quality metrics fail to detect them.
With the advent of Neural Machine Translation (NMT) systems, the MT output has reached unprecedented accuracy levels which resulted in the ubiquity of MT tools on almost all online platforms with multilingual content. However, NMT systems, like other state-of-the-art AI generative systems, are prone to errors that are deemed machine hallucinations. The problem with NMT hallucinations is that they are remarkably \textit{fluent} hallucinations. Since they are trained to produce grammatically correct utterances, NMT systems are capable of producing mistranslations that are too fluent to be recognised by both users of the MT tool, as well as by automatic quality metrics that are used to gauge their performance. In this paper, we introduce an authentic dataset of machine translation critical errors to point to the ethical and safety issues involved in the common use of MT. The dataset comprises mistranslations of Arabic mental health postings manually annotated with critical error types. We also show how the commonly used quality metrics do not penalise critical errors and highlight this as a critical issue that merits further attention from researchers.