Training neural networks with structured noise improves classification and generalization
This work addresses the challenge of optimizing neural network performance for memory and generalization tasks, with potential applications in modeling biological systems, but it appears incremental as it builds on existing noise-injection methods.
The paper tackled the problem of improving memory retrieval and generalization in neural networks by adding structured noise during training, achieving near-perfect retrieval and wide basins of attraction even with maximal noise.
The beneficial role of noise-injection in learning is a consolidated concept in the field of artificial neural networks, suggesting that even biological systems might take advantage of similar mechanisms to optimize their performance. The training-with-noise algorithm proposed by Gardner and collaborators is an emblematic example of a noise-injection procedure in recurrent networks, which can be used to model biological neural systems. We show how adding structure to noisy training data can substantially improve the algorithm performance, allowing the network to approach perfect retrieval of the memories and wide basins of attraction, even in the scenario of maximal injected noise. We also prove that the so-called Hebbian Unlearning rule coincides with the training-with-noise algorithm when noise is maximal and data are stable fixed points of the network dynamics.