Training large-scale ANNs on simulated resistive crossbar arrays
This work addresses a critical bottleneck in hardware acceleration for AI, though it is incremental as it builds on existing concepts with simulated improvements.
The paper tackles the challenge of training large-scale artificial neural networks on analog resistive crossbar arrays, which face issues like noise and limited precision, by proposing simple algorithmic measures that significantly improve simulated training performance on intermediate to large-scale networks.
Accelerating training of artificial neural networks (ANN) with analog resistive crossbar arrays is a promising idea. While the concept has been verified on very small ANNs and toy data sets (such as MNIST), more realistically sized ANNs and datasets have not yet been tackled. However, it is to be expected that device materials and hardware design constraints, such as noisy computations, finite number of resistive states of the device materials, saturating weight and activation ranges, and limited precision of analog-to-digital converters, will cause significant challenges to the successful training of state-of-the-art ANNs. By using analog hardware aware ANN training simulations, we here explore a number of simple algorithmic compensatory measures to cope with analog noise and limited weight and output ranges and resolutions, that dramatically improve the simulated training performances on RPU arrays on intermediately to large-scale ANNs.