Adaptive Variance-Penalized Continual Learning with Fisher Regularization
This work addresses the problem of catastrophic forgetting for neural networks in continual learning scenarios, representing an incremental improvement with a novel regularization mechanism.
The paper tackled catastrophic forgetting in neural networks by proposing an adaptive variance-penalized continual learning framework with Fisher regularization, which improved stability and performance on benchmarks like SplitMNIST and PermutedMNIST, achieving substantial gains over methods such as Variational Continual Learning and Elastic Weight Consolidation.
The persistent challenge of catastrophic forgetting in neural networks has motivated extensive research in continual learning . This work presents a novel continual learning framework that integrates Fisher-weighted asymmetric regularization of parameter variances within a variational learning paradigm. Our method dynamically modulates regularization intensity according to parameter uncertainty, achieving enhanced stability and performance. Comprehensive evaluations on standard continual learning benchmarks including SplitMNIST, PermutedMNIST, and SplitFashionMNIST demonstrate substantial improvements over existing approaches such as Variational Continual Learning and Elastic Weight Consolidation . The asymmetric variance penalty mechanism proves particularly effective in maintaining knowledge across sequential tasks while improving model accuracy. Experimental results show our approach not only boosts immediate task performance but also significantly mitigates knowledge degradation over time, effectively addressing the fundamental challenge of catastrophic forgetting in neural networks