Low-Precision Batch-Normalized Activations
This work addresses the problem of efficient deep neural network training for AI practitioners, though it appears incremental as it builds on prior low-precision methods.
The paper tackles the challenge of training very deep neural networks with low-precision arithmetic by introducing a quantization scheme for activations within batch-normalization modules, resulting in reduced memory and computational requirements with minimal accuracy loss.
Artificial neural networks can be trained with relatively low-precision floating-point and fixed-point arithmetic, using between one and 16 bits. Previous works have focused on relatively wide-but-shallow, feed-forward networks. We introduce a quantization scheme that is compatible with training very deep neural networks. Quantizing the network activations in the middle of each batch-normalization module can greatly reduce the amount of memory and computational power needed, with little loss in accuracy.