Dictionary Learning by Dynamical Neural Networks
This work addresses a specific bottleneck in implementing efficient, parallelizable neural networks for dictionary learning, but it is incremental as it builds on existing dynamical network concepts.
The authors tackled the challenge of computing gradients in dynamical neural networks under local information constraints by combining top-down feedback and contrastive learning, resulting in a provably correct method for l1-minimizing dictionary learning with numerical validation on several problems.
A dynamical neural network consists of a set of interconnected neurons that interact over time continuously. It can exhibit computational properties in the sense that the dynamical system's evolution and/or limit points in the associated state space can correspond to numerical solutions to certain mathematical optimization or learning problems. Such a computational system is particularly attractive in that it can be mapped to a massively parallel computer architecture for power and throughput efficiency, especially if each neuron can rely solely on local information (i.e., local memory). Deriving gradients from the dynamical network's various states while conforming to this last constraint, however, is challenging. We show that by combining ideas of top-down feedback and contrastive learning, a dynamical network for solving the l1-minimizing dictionary learning problem can be constructed, and the true gradients for learning are provably computable by individual neurons. Using spiking neurons to construct our dynamical network, we present a learning process, its rigorous mathematical analysis, and numerical results on several dictionary learning problems.