Learning Sparse, Distributed Representations using the Hebbian Principle
This work addresses the gap between neuroscience principles and machine learning methods, offering an incremental improvement for unsupervised representation learning in domains like image processing.
The paper tackled the limited use of the Hebbian principle in machine learning by developing competitive Hebbian learning algorithms to produce sparse, distributed neural codes, demonstrating superior performance over autoencoders in training deep convolutional nets on image datasets.
The "fire together, wire together" Hebbian model is a central principle for learning in neuroscience, but surprisingly, it has found limited applicability in modern machine learning. In this paper, we take a first step towards bridging this gap, by developing flavors of competitive Hebbian learning which produce sparse, distributed neural codes using online adaptation with minimal tuning. We propose an unsupervised algorithm, termed Adaptive Hebbian Learning (AHL). We illustrate the distributed nature of the learned representations via output entropy computations for synthetic data, and demonstrate superior performance, compared to standard alternatives such as autoencoders, in training a deep convolutional net on standard image datasets.