FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference
This work addresses the problem of slow deep learning inference for users of CPU-based systems, though it is incremental as it builds on existing quantization research.
The authors tackled the inefficiency of deep learning inference by developing FBGEMM, a high-performance kernel library for quantized inference on CPUs, which achieved over 2x performance gains compared to their production baseline.
Deep learning models typically use single-precision (FP32) floating point data types for representing activations and weights, but a slew of recent research work has shown that computations with reduced-precision data types (FP16, 16-bit integers, 8-bit integers or even 4- or 2-bit integers) are enough to achieve same accuracy as FP32 and are much more efficient. Therefore, we designed fbgemm, a high-performance kernel library, from ground up to perform high-performance quantized inference on current generation CPUs. fbgemm achieves efficiency by fusing common quantization operations with a high-performance gemm implementation and by shape- and size-specific kernel code generation at runtime. The library has been deployed at Facebook, where it delivers greater than 2x performance gains with respect to our current production baseline.