PARQ: Piecewise-Affine Regularized Quantization
This work addresses efficient model deployment for resource-constrained applications, but it is incremental as it builds on existing quantization methods.
The authors tackled the problem of quantization-aware training for large-scale models by introducing piecewise-affine regularization to cluster parameters toward discrete values, achieving competitive performance on vision tasks.
We develop a principled method for quantization-aware training (QAT) of large-scale machine learning models. Specifically, we show that convex, piecewise-affine regularization (PAR) can effectively induce the model parameters to cluster towards discrete values. We minimize PAR-regularized loss functions using an aggregate proximal stochastic gradient method (AProx) and prove that it has last-iterate convergence. Our approach provides an interpretation of the straight-through estimator (STE), a widely used heuristic for QAT, as the asymptotic form of PARQ. We conduct experiments to demonstrate that PARQ obtains competitive performance on convolution- and transformer-based vision tasks.