Linear Gradient Prediction with Control Variates
This work addresses training efficiency for neural network practitioners, but it appears incremental as it builds on existing control variate and Neural Tangent Kernel ideas.
The authors tackled the problem of reducing neural network training cost by using approximate predicted gradients instead of full gradients, and they empirically demonstrated efficacy on a vision transformer classification task.
We propose a new way of training neural networks, with the goal of reducing training cost. Our method uses approximate predicted gradients instead of the full gradients that require an expensive backward pass. We derive a control-variate-based technique that ensures our updates are unbiased estimates of the true gradient. Moreover, we propose a novel way to derive a predictor for the gradient inspired by the theory of the Neural Tangent Kernel. We empirically show the efficacy of the technique on a vision transformer classification task.