NITRO-D: Native Integer-only Training of Deep Convolutional Neural Networks
This addresses the need for efficient deep learning in resource-constrained environments where floating-point arithmetic is unavailable, offering a novel solution for integer-only training of CNNs, which is not incremental as it extends beyond prior work limited to MLPs.
The paper tackles the problem of training deep convolutional neural networks (CNNs) using only integer operations, which is challenging as existing methods typically rely on floating-point arithmetic. It introduces NITRO-D, a framework that achieves this, resulting in up to +5.96% improved test accuracy for integer-only MLPs and up to 76.14% reduced memory and 32.42% reduced energy consumption for CNNs compared to traditional floating-point methods.
Quantization is a pivotal technique for managing the growing computational and memory demands of Deep Neural Networks (DNNs). By reducing the number of bits used to represent weights and activations (typically from 32-bit Floating-Point (FP) to 16-bit or 8-bit integers), quantization reduces memory footprint, energy consumption, and execution time of DNNs. However, most existing methods typically target DNN inference, while training still relies on FP operations, limiting applicability in environments where FP arithmetic is unavailable. To date, only one prior work has addressed integer-only training, and only for Multi-Layer Perceptron (MLP) architectures. This paper introduces NITRO-D, a novel framework for training deep integer-only Convolutional Neural Networks (CNNs) that operate entirely in the integer domain for both training and inference. NITRO-D enables training of integer CNNs without requiring a separate quantization scheme. Specifically, it introduces a novel architecture that integrates multiple local-loss blocks, which include the proposed NITRO-Scaling layer and NITRO-ReLU activation function. The proposed framework also features a novel learning algorithm that employs local error signals and leverages IntegerSGD, an optimizer specifically designed for integer computations. NITRO-D is implemented as an open-source Python library. Extensive evaluations on state-of-the-art image recognition datasets demonstrate its effectiveness. For integer-only MLPs, NITRO-D improves test accuracy by up to +5.96% over the state-of-the-art. It also successfully trains integer-only CNNs, reducing memory requirements and energy consumption by up to 76.14% and 32.42%, respectively, compared to the traditional FP backpropagation algorithm.