Hebbian Memory-Augmented Recurrent Networks: Engram Neurons in Deep Learning
This addresses the problem of interpretability in deep learning for researchers and practitioners, though it appears incremental as performance is comparable to existing methods.
The paper tackles the problem of limited interpretability and long-range dependency modeling in recurrent networks by introducing the Engram Neural Network (ENN), which incorporates explicit memory with Hebbian plasticity and sparse retrieval. The result shows that ENN achieves comparable accuracy and perplexity to classical RNN variants on benchmarks like MNIST, CIFAR-10, and WikiText-103 while offering enhanced interpretability through observable memory dynamics.
Despite success across diverse tasks, current artificial recurrent network architectures rely primarily on implicit hidden-state memories, limiting their interpretability and ability to model long-range dependencies. In contrast, biological neural systems employ explicit, associative memory traces (i.e., engrams) strengthened through Hebbian synaptic plasticity and activated sparsely during recall. Motivated by these neurobiological insights, we introduce the Engram Neural Network (ENN), a novel recurrent architecture incorporating an explicit, differentiable memory matrix with Hebbian plasticity and sparse, attention-driven retrieval mechanisms. The ENN explicitly models memory formation and recall through dynamic Hebbian traces, improving transparency and interpretability compared to conventional RNN variants. We evaluate the ENN architecture on three canonical benchmarks: MNIST digit classification, CIFAR-10 image sequence modeling, and WikiText-103 language modeling. Our empirical results demonstrate that the ENN achieves accuracy and generalization performance broadly comparable to classical RNN, GRU, and LSTM architectures, with all models converging to similar accuracy and perplexity on the large-scale WikiText-103 task. At the same time, the ENN offers significant enhancements in interpretability through observable memory dynamics. Hebbian trace visualizations further reveal biologically plausible, structured memory formation processes, validating the potential of neuroscience-inspired mechanisms to inform the development of more interpretable and robust deep learning models.