Weighted Patterns as a Tool for Improving the Hopfield Model
This work addresses a specific limitation in neural network memory models for researchers in statistical physics and machine learning, offering an incremental improvement to the Hopfield model.
The authors tackled the problem of catastrophic memory destruction in the standard Hopfield model by generalizing it to assign weights to input patterns, representing their frequency. They found that this weighted model prevents catastrophic overfilling, allowing online learning without memory destruction, though memory capacity decreases compared to the standard model.
We generalize the standard Hopfield model to the case when a weight is assigned to each input pattern. The weight can be interpreted as the frequency of the pattern occurrence at the input of the network. In the framework of the statistical physics approach we obtain the saddle-point equation allowing us to examine the memory of the network. In the case of unequal weights our model does not lead to the catastrophic destruction of the memory due to its overfilling (that is typical for the standard Hopfield model). The real memory consists only of the patterns with weights exceeding a critical value that is determined by the weights distribution. We obtain the algorithm allowing us to find this critical value for an arbitrary distribution of the weights, and analyze in detail some particular weights distributions. It is shown that the memory decreases as compared to the case of the standard Hopfield model. However, in our model the network can learn online without the catastrophic destruction of the memory.