PQS (Prune, Quantize, and Sort): Low-Bitwidth Accumulation of Dot Products in Neural Network Computations
This work addresses energy efficiency and memory usage issues in neural network inference for deployment on resource-constrained devices, representing an incremental improvement over existing quantization methods.
The paper tackles the problem of high memory bandwidth and energy inefficiency from wide accumulators in quantized neural network dot products by introducing PQS, which combines pruning, quantization, and sorted accumulation to reduce accumulator bitwidth by 2.5x while maintaining accuracy comparable to floating-point baselines on image classification tasks.
We present PQS, which uses three techniques together - Prune, Quantize, and Sort - to achieve low-bitwidth accumulation of dot products in neural network computations. In conventional quantized (e.g., 8-bit) dot products, partial results are accumulated into wide (e.g., 32-bit) accumulators to avoid overflows when accumulating intermediate partial sums. However, such wide accumulators increase memory bandwidth usage and reduce energy efficiency. We show that iterative N:M pruning in floating point followed by quantization to 8 (or fewer) bits, and accumulation of partial products in a sorted order ("small to large") allows for accurate, compressed models with short dot product lengths that do not require wide accumulators. We design, analyze, and implement the PQS algorithm to eliminate accumulation overflows at inference time for several neural networks. Our method offers a 2.5x reduction in accumulator bitwidth while achieving model accuracy on par with floating-point baselines for multiple image classification tasks.