Training Dynamical Binary Neural Networks with Equilibrium Propagation
This enables on-chip learning for neuromorphic hardware with digital communication and limited memory, though it is incremental as it adapts an existing algorithm to binary constraints.
The paper tackled training binary neural networks (BNNs) with binary activations and weights using Equilibrium Propagation (EP), achieving near-full-precision accuracy on MNIST and CIFAR-10, with only a 1.9% drop on CIFAR-10 and within 1% loss on MNIST for fully connected architectures.
Equilibrium Propagation (EP) is an algorithm intrinsically adapted to the training of physical networks, thanks to the local updates of weights given by the internal dynamics of the system. However, the construction of such a hardware requires to make the algorithm compatible with existing neuromorphic CMOS technologies, which generally exploit digital communication between neurons and offer a limited amount of local memory. In this work, we demonstrate that EP can train dynamical networks with binary activations and weights. We first train systems with binary weights and full-precision activations, achieving an accuracy equivalent to that of full-precision models trained by standard EP on MNIST, and losing only 1.9% accuracy on CIFAR-10 with equal architecture. We then extend our method to the training of models with binary activations and weights on MNIST, achieving an accuracy within 1% of the full-precision reference for fully connected architectures and reaching the full-precision accuracy for convolutional architectures. Our extension of EP to binary networks opens new solutions for on-chip learning and provides a compact framework for training BNNs end-to-end with the same circuitry as for inference.