Low-Complexity LSTM Training and Inference with FloatSD8 Weight Representation
This work addresses efficiency improvements for LSTM models in resource-constrained environments, representing an incremental advancement by extending existing FloatSD technology from CNNs to RNNs.
The paper tackled the problem of reducing complexity in LSTM training and inference by applying FloatSD weight representation, quantizing gradients and activations to 8 bits, and reducing arithmetic precision to 16 bits, achieving fully preserved model accuracy and significantly reduced die area and power consumption in circuit implementation.
The FloatSD technology has been shown to have excellent performance on low-complexity convolutional neural networks (CNNs) training and inference. In this paper, we applied FloatSD to recurrent neural networks (RNNs), specifically long short-term memory (LSTM). In addition to FloatSD weight representation, we quantized the gradients and activations in model training to 8 bits. Moreover, the arithmetic precision for accumulations and the master copy of weights were reduced from 32 bits to 16 bits. We demonstrated that the proposed training scheme can successfully train several LSTM models from scratch, while fully preserving model accuracy. Finally, to verify the proposed method's advantage in implementation, we designed an LSTM neuron circuit and showed that it achieved significantly reduced die area and power consumption.