LightNN: Filling the Gap between Conventional Deep Neural Networks and Binarized Networks
This work addresses hardware designers' need for flexible accuracy-energy trade-offs in ASIC implementations, offering an incremental improvement over existing methods.
The paper tackles the trade-off between accuracy and energy efficiency in deep neural networks by proposing LightNN, which replaces multiplications with shifts and adds, achieving better accuracy than binarized networks with slight energy increase and more energy efficiency than conventional DNNs with minor accuracy loss, as verified on MNIST and CIFAR-10 datasets.
Application-specific integrated circuit (ASIC) implementations for Deep Neural Networks (DNNs) have been adopted in many systems because of their higher classification speed. However, although they may be characterized by better accuracy, larger DNNs require significant energy and area, thereby limiting their wide adoption. The energy consumption of DNNs is driven by both memory accesses and computation. Binarized Neural Networks (BNNs), as a trade-off between accuracy and energy consumption, can achieve great energy reduction, and have good accuracy for large DNNs due to its regularization effect. However, BNNs show poor accuracy when a smaller DNN configuration is adopted. In this paper, we propose a new DNN model, LightNN, which replaces the multiplications to one shift or a constrained number of shifts and adds. For a fixed DNN configuration, LightNNs have better accuracy at a slight energy increase than BNNs, yet are more energy efficient with only slightly less accuracy than conventional DNNs. Therefore, LightNNs provide more options for hardware designers to make trade-offs between accuracy and energy. Moreover, for large DNN configurations, LightNNs have a regularization effect, making them better in accuracy than conventional DNNs. These conclusions are verified by experiment using the MNIST and CIFAR-10 datasets for different DNN configurations.