Fortuitous Forgetting in Connectionist Networks
This work addresses a foundational issue in machine learning by rethinking forgetting as a tool for enhancing neural network training, offering insights for iterative algorithms in domains like image classification and language emergence.
The paper tackles the problem of forgetting in neural networks by proposing that it can be beneficial, introducing a 'forget-and-relearn' paradigm to shape learning trajectories, which unifies existing iterative training algorithms and leads to performance improvements.
Forgetting is often seen as an unwanted characteristic in both human and machine learning. However, we propose that forgetting can in fact be favorable to learning. We introduce "forget-and-relearn" as a powerful paradigm for shaping the learning trajectories of artificial neural networks. In this process, the forgetting step selectively removes undesirable information from the model, and the relearning step reinforces features that are consistently useful under different conditions. The forget-and-relearn framework unifies many existing iterative training algorithms in the image classification and language emergence literature, and allows us to understand the success of these algorithms in terms of the disproportionate forgetting of undesirable information. We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations. Insights from our analysis provide a coherent view on the dynamics of iterative training in neural networks and offer a clear path towards performance improvements.