Bayesian Metaplasticity from Synaptic Uncertainty
This addresses the problem of forgetting in continual learning for AI systems, presenting an incremental improvement over existing methods.
The paper tackles catastrophic forgetting in neural networks for lifelong learning by introducing MESU, which uses synaptic uncertainty to retain information, achieving maintained performance across 100 permuted MNIST tasks without task boundaries.
Catastrophic forgetting remains a challenge for neural networks, especially in lifelong learning scenarios. In this study, we introduce MEtaplasticity from Synaptic Uncertainty (MESU), inspired by metaplasticity and Bayesian inference principles. MESU harnesses synaptic uncertainty to retain information over time, with its update rule closely approximating the diagonal Newton's method for synaptic updates. Through continual learning experiments on permuted MNIST tasks, we demonstrate MESU's remarkable capability to maintain learning performance across 100 tasks without the need of explicit task boundaries.