mT5: A massively multilingual pre-trained text-to-text transformer
This work addresses the need for scalable and effective multilingual models in NLP, though it is incremental as it builds directly on the T5 framework.
The paper tackles the problem of extending the T5 model to support multilingual NLP tasks by introducing mT5, a pre-trained text-to-text transformer covering 101 languages, which achieves state-of-the-art performance on various multilingual benchmarks.
The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.