Continual-learning for Modelling Low-Resource Languages from Large Language Models
This addresses the problem of catastrophic forgetting for researchers and practitioners working on low-resource language modeling, though it appears incremental as it builds on existing continual learning and adapter techniques.
The paper tackles catastrophic forgetting when adapting large language models to small language models for low-resource languages, proposing a continual learning strategy with POS-based code-switching and replay adapters that shows success in vision-language tasks like visual question answering and language modeling.
Modelling a language model for a multi-lingual scenario includes several potential challenges, among which catastrophic forgetting is the major challenge. For example, small language models (SLM) built for low-resource languages by adapting large language models (LLMs) pose the challenge of catastrophic forgetting. This work proposes to employ a continual learning strategy using parts-of-speech (POS)-based code-switching along with a replay adapter strategy to mitigate the identified gap of catastrophic forgetting while training SLM from LLM. Experiments conducted on vision language tasks such as visual question answering and language modelling task exhibits the success of the proposed architecture.