Optimization without Backpropagation
This work addresses the problem of efficient optimization methods for machine learning practitioners, but it is incremental as it builds on existing forward gradient techniques.
The paper tackled the challenge of using forward gradients as an alternative to backpropagation for optimization, finding that optimization in high dimensions is difficult with this approach, as supported by experiments on test functions.
Forward gradients have been recently introduced to bypass backpropagation in autodifferentiation, while retaining unbiased estimators of true gradients. We derive an optimality condition to obtain best approximating forward gradients, which leads us to mathematical insights that suggest optimization in high dimension is challenging with forward gradients. Our extensive experiments on test functions support this claim.