Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control
This addresses the problem of catastrophic forgetting for neural networks in continual learning, offering an incremental improvement over existing methods.
The paper tackled catastrophic forgetting in neural networks by proposing neuron-level plasticity control (NPC), which estimates neuron importance and applies lower learning rates to consolidate them, resulting in substantially more effective performance compared to connection-level approaches on incremental MNIST and CIFAR100 datasets.
To address the issue of catastrophic forgetting in neural networks, we propose a novel, simple, and effective solution called neuron-level plasticity control (NPC). While learning a new task, the proposed method preserves the knowledge for the previous tasks by controlling the plasticity of the network at the neuron level. NPC estimates the importance value of each neuron and consolidates important \textit{neurons} by applying lower learning rates, rather than restricting individual connection weights to stay close to certain values. The experimental results on the incremental MNIST (iMNIST) and incremental CIFAR100 (iCIFAR100) datasets show that neuron-level consolidation is substantially more effective compared to the connection-level consolidation approaches.