Mitigating Asymmetric Nonlinear Weight Update Effects in Hardware Neural Network based on Analog Resistive Synapse
This work addresses a critical hardware-specific bottleneck for implementing efficient online training in neural networks, offering incremental improvements for domain applications like edge computing.
The paper tackled the problem of asymmetric nonlinear weight updates in hardware neural networks using analog resistive synapses, which hinder online training, by co-optimizing hardware-applicable deep-learning algorithms and engineering activation functions with a threshold weight update scheme, resulting in improved classification accuracy on MNIST to 87.8% with 6-bit and 94.8% with 8-bit analog synapses under high asymmetric nonlinearity.
Asymmetric nonlinear weight update is considered as one of the major obstacles for realizing hardware neural networks based on analog resistive synapses because it significantly compromises the online training capability. This paper provides new solutions to this critical issue through co-optimization with the hardware-applicable deep-learning algorithms. New insights on engineering activation functions and a threshold weight update scheme effectively suppress the undesirable training noise induced by inaccurate weight update. We successfully trained a two-layer perceptron network online and improved the classification accuracy of MNIST handwritten digit dataset to 87.8/94.8% by using 6-bit/8-bit analog synapses, respectively, with extremely high asymmetric nonlinearity.