A Call for Clarity in Reporting BLEU Scores
This addresses a reproducibility and comparability problem for machine translation researchers, though it is incremental as it builds on existing metric practices.
The paper tackles the inconsistency in reporting BLEU scores in machine translation, quantifying variations up to 1.8 due to unreported parameters, and proposes adopting the WMT BLEU scheme with a new tool, SacreBLEU, to standardize comparisons.
The field of machine translation faces an under-recognized problem because of inconsistency in the reporting of scores from its dominant metric. Although people refer to "the" BLEU score, BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters. These parameters are often not reported or are hard to find, and consequently, BLEU scores between papers cannot be directly compared. I quantify this variation, finding differences as high as 1.8 between commonly used configurations. The main culprit is different tokenization and normalization schemes applied to the reference. Pointing to the success of the parsing community, I suggest machine translation researchers settle upon the BLEU scheme used by the annual Conference on Machine Translation (WMT), which does not allow for user-supplied reference processing, and provide a new tool, SacreBLEU, to facilitate this.