AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model
This work addresses the efficiency and performance of large language models for few-shot learning across multiple languages, offering a compelling alternative to decoder-only models.
The paper tackles the problem of few-shot learning by training a 20 billion parameter multilingual seq2seq model, AlexaTM 20B, which achieves state-of-the-art performance on tasks like 1-shot summarization and machine translation, outperforming larger models like PaLM 540B and GPT3 175B in various benchmarks.
In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, are more efficient few-shot learners than decoder-only models on various tasks. In particular, we train a 20 billion parameter multilingual seq2seq model called Alexa Teacher Model (AlexaTM 20B) and show that it achieves state-of-the-art (SOTA) performance on 1-shot summarization tasks, outperforming a much larger 540B PaLM decoder model. AlexaTM 20B also achieves SOTA in 1-shot machine translation, especially for low-resource languages, across almost all language pairs supported by the model (Arabic, English, French, German, Hindi, Italian, Japanese, Marathi, Portuguese, Spanish, Tamil, and Telugu) on Flores-101 dataset. We also show in zero-shot setting, AlexaTM 20B outperforms GPT3 (175B) on SuperGLUE and SQuADv2 datasets and provides SOTA performance on multilingual tasks such as XNLI, XCOPA, Paws-X, and XWinograd. Overall, our results present a compelling case for seq2seq models as a powerful alternative to decoder-only models for Large-scale Language Model (LLM) training.