PPSEBM: An Energy-Based Model with Progressive Parameter Selection for Continual Learning
This addresses the problem of forgetting past knowledge in continual learning for NLP practitioners, offering a robust solution.
The paper tackled catastrophic forgetting in continual learning for NLP by introducing PPSEBM, which combines an Energy-Based Model with Progressive Parameter Selection, and it outperformed state-of-the-art methods on diverse benchmarks.
Continual learning remains a fundamental challenge in machine learning, requiring models to learn from a stream of tasks without forgetting previously acquired knowledge. A major obstacle in this setting is catastrophic forgetting, where performance on earlier tasks degrades as new tasks are learned. In this paper, we introduce PPSEBM, a novel framework that integrates an Energy-Based Model (EBM) with Progressive Parameter Selection (PPS) to effectively address catastrophic forgetting in continual learning for natural language processing tasks. In PPSEBM, progressive parameter selection allocates distinct, task-specific parameters for each new task, while the EBM generates representative pseudo-samples from prior tasks. These generated samples actively inform and guide the parameter selection process, enhancing the model's ability to retain past knowledge while adapting to new tasks. Experimental results on diverse NLP benchmarks demonstrate that PPSEBM outperforms state-of-the-art continual learning methods, offering a promising and robust solution to mitigate catastrophic forgetting.