Conditional Computation for Continual Learning
This work addresses catastrophic forgetting for continual learning systems, offering an incremental improvement by refining rehearsal methods based on parameter sharing analysis.
The paper tackles catastrophic forgetting in neural networks by analyzing parameter sharing under conditional computation and introduces a clipped maxout network that partially shares parameters, enabling targeted rehearsal on interfered examples to prevent forgetting, achieving effective performance in online non-stationary setups.
Catastrophic forgetting of connectionist neural networks is caused by the global sharing of parameters among all training examples. In this study, we analyze parameter sharing under the conditional computation framework where the parameters of a neural network are conditioned on each input example. At one extreme, if each input example uses a disjoint set of parameters, there is no sharing of parameters thus no catastrophic forgetting. At the other extreme, if the parameters are the same for every example, it reduces to the conventional neural network. We then introduce a clipped version of maxout networks which lies in the middle, i.e. parameters are shared partially among examples. Based on the parameter sharing analysis, we can locate a limited set of examples that are interfered when learning a new example. We propose to perform rehearsal on this set to prevent forgetting, which is termed as conditional rehearsal. Finally, we demonstrate the effectiveness of the proposed method in an online non-stationary setup, where updates are made after each new example and the distribution of the received example shifts over time.