On Robust Incremental Learning over Many Multilingual Steps
This work addresses incremental learning challenges for scenarios with privacy constraints, though it appears incremental in nature.
The paper tackles the problem of catastrophic forgetting in incremental learning by proposing a method that enables robust learning over dozens of fine-tuning steps using multilingual data, achieving continued improvement for up to fifty steps without retaining previous training data.
Recent work in incremental learning has introduced diverse approaches to tackle catastrophic forgetting from data augmentation to optimized training regimes. However, most of them focus on very few training steps. We propose a method for robust incremental learning over dozens of fine-tuning steps using data from a variety of languages. We show that a combination of data-augmentation and an optimized training regime allows us to continue improving the model even for as many as fifty training steps. Crucially, our augmentation strategy does not require retaining access to previous training data and is suitable in scenarios with privacy constraints.