Can a Hebbian-like learning rule be avoiding the curse of dimensionality in sparse distributed data?
This addresses a challenge in brain-inspired machine learning for sparse data, but it is incremental as it builds on existing Hebbian and RBM methods.
The paper tackles the problem of the curse of dimensionality in sparse distributed data by proposing that a Hebbian-like learning rule in Restricted Boltzmann Machines can avoid it by ignoring zeros, and experiments show that this approach achieves good generalization performance while backpropagation-trained networks overfit.
It is generally assumed that the brain uses something akin to sparse distributed representations. These representations, however, are high-dimensional and consequently they affect classification performance of traditional Machine Learning models due to "the curse of dimensionality". In tasks for which there is a vast amount of labeled data, Deep Networks seem to solve this issue with many layers and a non-Hebbian backpropagation algorithm. The brain, however, seems to be able to solve the problem with few layers. In this work, we hypothesize that this happens by using Hebbian learning. Actually, the Hebbian-like learning rule of Restricted Boltzmann Machines learns the input patterns asymmetrically. It exclusively learns the correlation between non-zero values and ignores the zeros, which represent the vast majority of the input dimensionality. By ignoring the zeros "the curse of dimensionality" problem can be avoided. To test our hypothesis, we generated several sparse datasets and compared the performance of a Restricted Boltzmann Machine classifier with some Backprop-trained networks. The experiments using these codes confirm our initial intuition as the Restricted Boltzmann Machine shows a good generalization performance, while the Neural Networks trained with the backpropagation algorithm overfit the training data.