Quantization-Aware Regularizers for Deep Neural Networks Compression
This addresses the problem of deploying large models on resource-constrained devices by improving quantization methods, though it appears incremental as it builds on existing quantization techniques.
The paper tackles the accuracy drop in deep neural network compression via weight quantization by introducing per-layer regularization terms that drive weights to form clusters during training, integrating quantization awareness into optimization, which reduces accuracy loss while preserving compression potential, with experiments on CIFAR-10 showing effectiveness.
Deep Neural Networks reached state-of-the-art performance across numerous domains, but this progress has come at the cost of increasingly large and over-parameterized models, posing serious challenges for deployment on resource-constrained devices. As a result, model compression has become essential, and -- among compression techniques -- weight quantization is largely used and particularly effective, yet it typically introduces a non-negligible accuracy drop. However, it is usually applied to already trained models, without influencing how the parameter space is explored during the learning phase. In contrast, we introduce per-layer regularization terms that drive weights to naturally form clusters during training, integrating quantization awareness directly into the optimization process. This reduces the accuracy loss typically associated with quantization methods while preserving their compression potential. Furthermore, in our framework quantization representatives become network parameters, marking, to the best of our knowledge, the first approach to embed quantization parameters directly into the backpropagation procedure. Experiments on CIFAR-10 with AlexNet and VGG16 models confirm the effectiveness of the proposed strategy.