DiaBLa: A Corpus of Bilingual Spontaneous Written Dialogues for Machine Translation
This provides a unique resource for evaluating MT models and analyzing MT-mediated communication, but it is incremental as it focuses on a specific domain of informal dialogues.
The authors introduced DiaBLa, a new English-French test set with 144 spontaneous dialogues (over 5,700 sentences) for evaluating machine translation in informal written dialogues, and found that participant judgments revealed perceptible differences in quality between two neural MT systems.
We present a new English-French test set for the evaluation of Machine Translation (MT) for informal, written bilingual dialogue. The test set contains 144 spontaneous dialogues (5,700+ sentences) between native English and French speakers, mediated by one of two neural MT systems in a range of role-play settings. The dialogues are accompanied by fine-grained sentence-level judgments of MT quality, produced by the dialogue participants themselves, as well as by manually normalised versions and reference translations produced a posteriori. The motivation for the corpus is two-fold: to provide (i) a unique resource for evaluating MT models, and (ii) a corpus for the analysis of MT-mediated communication. We provide a preliminary analysis of the corpus to confirm that the participants' judgments reveal perceptible differences in MT quality between the two MT systems used.