Continual Learning for Text Classification with Information Disentanglement Based Regularization
This work addresses the challenge of enabling NLP models to generalize better across tasks in continual learning, representing an incremental improvement over existing methods.
The paper tackles the problem of poor generalization to new tasks in continual learning for text classification by proposing an information disentanglement based regularization method, which achieves state-of-the-art performance on large-scale benchmarks.
Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https://github.com/GT-SALT/IDBR.