Rescaling-Aware Training for Efficient Deployment of Deep Learning Models on Full-Integer Hardware
This work addresses energy and cost efficiency for AI inference in resource-constrained embedded systems, representing an incremental improvement over existing quantization-aware training methods.
The paper tackles the problem of integer rescaling being a costly operation in integer-only AI inference by proposing a method to reduce rescaling cost through stronger quantization of rescale multiplicands without accuracy loss, and introduces Rescale-Aware Training for ultra-low bit-width rescaling, preserving full accuracy with 8x reduced rescaler widths.
Integer AI inference significantly reduces computational complexity in embedded systems. Quantization-aware training (QAT) helps mitigate accuracy degradation associated with post-training quantization but still overlooks the impact of integer rescaling during inference, which is a hardware costly operation in integer-only AI inference. This work shows that rescaling cost can be dramatically reduced post-training, by applying a stronger quantization to the rescale multiplicands at no model-quality loss. Furthermore, we introduce Rescale-Aware Training, a fine tuning method for ultra-low bit-width rescaling multiplicands. Experiments show that even with 8x reduced rescaler widths, the full accuracy is preserved through minimal incremental retraining. This enables more energy-efficient and cost-efficient AI inference for resource-constrained embedded systems.