Dynamic Precision Analog Computing for Neural Networks
This work addresses energy efficiency challenges in analog computing for neural networks, offering a method to optimize precision dynamically without retraining, which is incremental but impactful for specific hardware applications.
The paper tackles the problem of analog computing's precision limitations due to noise by proposing dynamic precision architectures that repeat operations to reduce noise impact, enabling programmable trade-offs between precision and performance metrics like energy efficiency, resulting in energy consumption reductions of up to 89% for Resnet50 and 24% for BERT with minimal accuracy degradation.
Analog electronic and optical computing exhibit tremendous advantages over digital computing for accelerating deep learning when operations are executed at low precision. In this work, we derive a relationship between analog precision, which is limited by noise, and digital bit precision. We propose extending analog computing architectures to support varying levels of precision by repeating operations and averaging the result, decreasing the impact of noise. Such architectures enable programmable tradeoffs between precision and other desirable performance metrics such as energy efficiency or throughput. To utilize dynamic precision, we propose a method for learning the precision of each layer of a pre-trained model without retraining network weights. We evaluate this method on analog architectures subject to a variety of noise sources such as shot noise, thermal noise, and weight noise and find that employing dynamic precision reduces energy consumption by up to 89% for computer vision models such as Resnet50 and by 24% for natural language processing models such as BERT. In one example, we apply dynamic precision to a shot-noise limited homodyne optical neural network and simulate inference at an optical energy consumption of 2.7 aJ/MAC for Resnet50 and 1.6 aJ/MAC for BERT with <2% accuracy degradation.