Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
This work assesses the translation capabilities of a large multilingual model, providing insights for NLP researchers and practitioners working on cross-lingual tasks, though it is incremental as it applies existing evaluation methods to a new model.
The study evaluated BLOOM's machine translation performance across multiple datasets and language pairs, finding that 0-shot performance had issues like overgeneration and wrong language output, but few-shot settings significantly improved results, achieving very good performance for many language pairs.
The NLP community recently saw the release of a new large open-access multilingual language model, BLOOM (BigScience et al., 2022) covering 46 languages. We focus on BLOOM's multilingual ability by evaluating its machine translation performance across several datasets (WMT, Flores-101 and DiaBLa) and language pairs (high- and low-resourced). Our results show that 0-shot performance suffers from overgeneration and generating in the wrong language, but this is greatly improved in the few-shot setting, with very good results for a number of language pairs. We study several aspects including prompt design, model sizes, cross-lingual transfer and the use of discursive context.