PIPE : Parallelized Inference Through Post-Training Quantization Ensembling of Residual Expansions
This addresses the need for flexible, efficient inference methods for AI applications on diverse hardware, though it appears incremental as it builds on existing quantization techniques.
The paper tackles the problem of high inference cost in deep neural networks by proposing PIPE, a data-free quantization method that adapts to various devices and bit-widths, achieving superior performance across vision and NLP tasks, architectures, and bit-widths from int8 to ternary.
Deep neural networks (DNNs) are ubiquitous in computer vision and natural language processing, but suffer from high inference cost. This problem can be addressed by quantization, which consists in converting floating point perations into a lower bit-width format. With the growing concerns on privacy rights, we focus our efforts on data-free methods. However, such techniques suffer from their lack of adaptability to the target devices, as a hardware typically only support specific bit widths. Thus, to adapt to a variety of devices, a quantization method shall be flexible enough to find good accuracy v.s. speed trade-offs for every bit width and target device. To achieve this, we propose PIPE, a quantization method that leverages residual error expansion, along with group sparsity and an ensemble approximation for better parallelization. PIPE is backed off by strong theoretical guarantees and achieves superior performance on every benchmarked application (from vision to NLP tasks), architecture (ConvNets, transformers) and bit-width (from int8 to ternary quantization).