Neural Network Compression Framework for fast model inference
This work addresses the need for faster model inference on hardware like CPUs, GPUs, and accelerators, but it is incremental as it leverages existing compression methods.
The authors tackled the problem of neural network inference speed by developing a compression framework that integrates sparsity, quantization, and binarization, resulting in accelerated inference time while maintaining original accuracy across various models.
In this work we present a new framework for neural networks compression with fine-tuning, which we called Neural Network Compression Framework (NNCF). It leverages recent advances of various network compression methods and implements some of them, such as sparsity, quantization, and binarization. These methods allow getting more hardware-friendly models which can be efficiently run on general-purpose hardware computation units (CPU, GPU) or special Deep Learning accelerators. We show that the developed methods can be successfully applied to a wide range of models to accelerate the inference time while keeping the original accuracy. The framework can be used within the training samples, which are supplied with it, or as a standalone package that can be seamlessly integrated into the existing training code with minimal adaptations. Currently, a PyTorch version of NNCF is available as a part of OpenVINO Training Extensions at https://github.com/openvinotoolkit/nncf.