Exploring Pretraining via Active Forgetting for Improving Cross Lingual Transfer for Decoder Language Models
This addresses the challenge of adapting LLMs to non-English languages, which is incremental as it builds on prior work for encoder-only models.
The paper tackles the problem of limited cross-lingual transfer in decoder-only large language models (LLMs) by proposing a pretraining strategy using active forgetting, which results in better multilingual representations and improved performance in downstream tasks.
Large Language Models (LLMs) demonstrate exceptional capabilities in a multitude of NLP tasks. However, the efficacy of such models to languages other than English is often limited. Prior works have shown that encoder-only models such as BERT or XLM-RoBERTa show impressive cross lingual transfer of their capabilities from English to other languages. In this work, we propose a pretraining strategy that uses active forgetting to achieve similar cross lingual transfer in decoder-only LLMs. We show that LLMs pretrained with active forgetting are highly effective when adapting to new and unseen languages. Through extensive experimentation, we find that LLMs pretrained with active forgetting are able to learn better multilingual representations which translates to better performance in many downstream tasks.