How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks
This work provides incremental insights into continual learning mechanisms, potentially aiding researchers in designing more robust AI systems for sequential tasks.
The study investigated how weight resampling (zapping) and optimizer choice affect learning and forgetting dynamics in neural networks under continual and few-shot transfer learning settings, finding that zapping accelerates recovery from domain shifts and optimizer selection influences task synergy/interference patterns.
Recent work in continual learning has highlighted the beneficial effect of resampling weights in the last layer of a neural network (``zapping"). Although empirical results demonstrate the effectiveness of this approach, the underlying mechanisms that drive these improvements remain unclear. In this work, we investigate in detail the pattern of learning and forgetting that take place inside a convolutional neural network when trained in challenging settings such as continual learning and few-shot transfer learning, with handwritten characters and natural images. Our experiments show that models that have undergone zapping during training more quickly recover from the shock of transferring to a new domain. Furthermore, to better observe the effect of continual learning in a multi-task setting we measure how each individual task is affected. This shows that, not only zapping, but the choice of optimizer can also deeply affect the dynamics of learning and forgetting, causing complex patterns of synergy/interference between tasks to emerge when the model learns sequentially at transfer time.