Synaptic metaplasticity in binarized neural networks
This addresses the problem of catastrophic forgetting for researchers and practitioners using binarized neural networks, but it is incremental as it applies an existing biological concept to a specific network type.
The paper tackled catastrophic forgetting in artificial neural networks by transferring the concept of synaptic metaplasticity from neuroscience to binarized neural networks, resulting in reduced forgetting.
Unlike the brain, artificial neural networks, including state-of-the-art deep neural networks for computer vision, are subject to "catastrophic forgetting": they rapidly forget the previous task when trained on a new one. Neuroscience suggests that biological synapses avoid this issue through the process of synaptic consolidation and metaplasticity: the plasticity itself changes upon repeated synaptic events. In this work, we show that this concept of metaplasticity can be transferred to a particular type of deep neural networks, binarized neural networks, to reduce catastrophic forgetting.