Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients
This addresses efficiency and scalability issues for deep learning practitioners, though it appears incremental as an enhancement to existing forward gradient methods.
The paper tackles the computational expense and parallelization limitations of backpropagation in neural networks by analyzing multi-tangent forward gradients, showing that increasing tangents improves approximation quality and optimization performance across various tasks.
The gradients used to train neural networks are typically computed using backpropagation. While an efficient way to obtain exact gradients, backpropagation is computationally expensive, hinders parallelization, and is biologically implausible. Forward gradients are an approach to approximate the gradients from directional derivatives along random tangents computed by forward-mode automatic differentiation. So far, research has focused on using a single tangent per step. This paper provides an in-depth analysis of multi-tangent forward gradients and introduces an improved approach to combining the forward gradients from multiple tangents based on orthogonal projections. We demonstrate that increasing the number of tangents improves both approximation quality and optimization performance across various tasks.