Fixed-point optimization of deep neural networks with adaptive step size retraining
This work addresses the need for low-power hardware implementations of deep neural networks, though it appears incremental as it builds on existing quantization and retraining methods.
The paper tackled the problem of optimizing deep neural networks for fixed-point hardware by proposing an algorithm that dynamically estimates quantization step size during retraining and testing a gradual quantization scheme, achieving good performance with 2- or 3-bit precision across feed-forward, convolutional, and recurrent neural networks.
Fixed-point optimization of deep neural networks plays an important role in hardware based design and low-power implementations. Many deep neural networks show fairly good performance even with 2- or 3-bit precision when quantized weights are fine-tuned by retraining. We propose an improved fixedpoint optimization algorithm that estimates the quantization step size dynamically during the retraining. In addition, a gradual quantization scheme is also tested, which sequentially applies fixed-point optimizations from high- to low-precision. The experiments are conducted for feed-forward deep neural networks (FFDNNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs).