StackNet: Stacking Parameters for Continual learning
This addresses the problem of incremental data accumulation in neural networks for researchers and practitioners, though it is incremental as it builds on prior work like PackNet.
The paper tackles catastrophic forgetting in continual learning by proposing StackNet, a method that stacks parameters and uses an index module to assign tasks to specific network portions, achieving competitive performance without degrading previously learned tasks.
Training a neural network for a classification task typically assumes that the data to train are given from the beginning. However, in the real world, additional data accumulate gradually and the model requires additional training without accessing the old training data. This usually leads to the catastrophic forgetting problem which is inevitable for the traditional training methodology of neural networks. In this paper, we propose a continual learning method that is able to learn additional tasks while retaining the performance of previously learned tasks by stacking parameters. Composed of two complementary components, the index module and the StackNet, our method estimates the index of the corresponding task for an input sample with the index module and utilizes a particular portion of StackNet with this index. The StackNet guarantees no degradation in the performance of the previously learned tasks and the index module shows high confidence in finding the origin of an input sample. Compared to the previous work of PackNet, our method is competitive and highly intuitive.