BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights
This work addresses the challenge of efficient model deployment for resource-constrained devices by enhancing quantization training, though it is incremental as it builds on existing relaxation techniques.
The authors tackled the problem of training deep neural networks with quantized weights by proposing BinaryRelax, a two-phase algorithm that relaxes quantization constraints during training and gradually enforces them, resulting in improved accuracy over state-of-the-art methods on CIFAR and ImageNet datasets.
We propose BinaryRelax, a simple two-phase algorithm, for training deep neural networks with quantized weights. The set constraint that characterizes the quantization of weights is not imposed until the late stage of training, and a sequence of \emph{pseudo} quantized weights is maintained. Specifically, we relax the hard constraint into a continuous regularizer via Moreau envelope, which turns out to be the squared Euclidean distance to the set of quantized weights. The pseudo quantized weights are obtained by linearly interpolating between the float weights and their quantizations. A continuation strategy is adopted to push the weights towards the quantized state by gradually increasing the regularization parameter. In the second phase, exact quantization scheme with a small learning rate is invoked to guarantee fully quantized weights. We test BinaryRelax on the benchmark CIFAR and ImageNet color image datasets to demonstrate the superiority of the relaxed quantization approach and the improved accuracy over the state-of-the-art training methods. Finally, we prove the convergence of BinaryRelax under an approximate orthogonality condition.