Gradient Estimation Methods of Approximate Multipliers for High-Accuracy Retraining of Deep Learning Models
This work addresses the need for more efficient and accurate retraining in deep learning accelerators, offering incremental improvements over prior gradient estimation methods.
The paper tackles the problem of suboptimal retraining accuracy in deep learning models using approximate multipliers by proposing two gradient estimation methods, LUT-2D and LUT-1D, which improve retraining accuracy by up to 3.83% on CIFAR-10 and 23.69% on ImageNet compared to existing approaches.
Approximate multipliers (AppMults) are widely used in deep learning accelerators to reduce their area, delay, and power consumption. However, AppMults introduce arithmetic errors into deep learning models, necessitating a retraining process to recover accuracy. A key step in retraining is computing the gradient of the AppMult, i.e., the partial derivative of the approximate product with respect to each input operand. Existing approaches typically estimate this gradient using that of the accurate multiplier (AccMult), which can lead to suboptimal retraining results. To address this, we propose two methods to obtain more precise gradients of AppMults. The first, called LUT-2D, characterizes the AppMult gradient with 2-dimensional lookup tables (LUTs), providing fine-grained estimation and achieving the highest retraining accuracy. The second, called LUT-1D, is a compact and more efficient variant that stores gradient values in 1-dimensional LUTs, achieving comparable retraining accuracy with shorter runtime. Experimental results show that on CIFAR-10 with convolutional neural networks, our LUT-2D and LUT-1D methods improve retraining accuracy by 3.83% and 3.72% on average, respectively. On ImageNet with vision transformer models, our LUT-1D method improves retraining accuracy by 23.69% on average, compared to a state-of-the-art retraining framework.