Hopfield model with planted patterns: a teacher-student self-supervised learning model
This work connects classical Hopfield networks to modern machine learning, offering insights into self-supervised learning mechanisms, though it appears incremental as an adaptation of existing models.
The authors formulated a teacher-student self-supervised learning problem using a generalized Hopfield model with Boltzmann machines, analyzing performance via phase diagrams based on dataset size, noise, and regularization. They found that with a small informative dataset, learning occurs through memorization, while with noisy data, a critical number of examples enables a generalization regime.
While Hopfield networks are known as paradigmatic models for memory storage and retrieval, modern artificial intelligence systems mainly stand on the machine learning paradigm. We show that it is possible to formulate a teacher-student self-supervised learning problem with Boltzmann machines in terms of a suitable generalization of the Hopfield model with structured patterns, where the spin variables are the machine weights and patterns correspond to the training set's examples. We analyze the learning performance by studying the phase diagram in terms of the training set size, the dataset noise and the inference temperature (i.e. the weight regularization). With a small but informative dataset the machine can learn by memorization. With a noisy dataset, an extensive number of examples above a critical threshold is needed. In this regime the memory storage limits of the system becomes an opportunity for the occurrence of a learning regime in which the system can generalize.