Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfitting
This work addresses overfitting and generalization issues in neural networks, offering strategies like regularization and early-stopping, but it is incremental as it builds on existing Hopfield-like setups.
The authors tackled the problem of generalization and overfitting in attractor neural networks by optimizing network parameters via gradient descent on a regularized loss function, resulting in Hebbian kernels revised through unlearning, with analytical and numerical experiments showing regimes of overfitting, failure, and success as dataset parameters vary.
In this work we approach attractor neural networks from a machine learning perspective: we look for optimal network parameters by applying a gradient descent over a regularized loss function. Within this framework, the optimal neuron-interaction matrices turn out to be a class of matrices which correspond to Hebbian kernels revised by a reiterated unlearning protocol. Remarkably, the extent of such unlearning is proved to be related to the regularization hyperparameter of the loss function and to the training time. Thus, we can design strategies to avoid overfitting that are formulated in terms of regularization and early-stopping tuning. The generalization capabilities of these attractor networks are also investigated: analytical results are obtained for random synthetic datasets, next, the emerging picture is corroborated by numerical experiments that highlight the existence of several regimes (i.e., overfitting, failure and success) as the dataset parameters are varied.