A method of using RSVD in residual calculation of LowBit GEMM
This addresses accuracy degradation in low-precision computations for deep learning and hardware applications, offering a data-free method with minimal speed reduction.
The paper tackles the problem of computational errors in low-precision matrix multiplication by proposing a low-rank residuals quantized method, which reduces error by 1-2 orders of magnitude and achieves 61.8% ImageNet Top-1 accuracy in ResNet-50 with 4-bit quantization compared to 8.3% for direct quantization.
The advancements of hardware technology in recent years has brought many possibilities for low-precision applications. However, the use of low precision can introduce significant computational errors, posing a considerable challenge to maintaining the computational accuracy. We propose low-rank residuals quantized matrix multiplication(LRQMM) method which introduces low-rank approximation in residual compensation for dense low precision quantization matrix multiplication. It can bring several times accuracy improvement with only BLAS-2 level extra time overhead. Moreover, LRQMM is a completely data-free quantization method that does not require additional data for pre-training. And it only works with low precision GEMM operator, which is easy to couple with other methods. Through experimentation, LRQMM can reduce the error of direct quantized matrix multiplication by 1~2 orders of magnitude, when dealing with larger matrix sizes, the computational speed is only reduced by approximately 20\%. In deep learning networks, LRQMM-4bit achieves 61.8% ImageNet Top-1 accuracy in Resnet-50, while the Direct Quant accuracy is only 8.3%.