Is Backpropagation Optimal? When Synthetic Gradients Improve Sample Efficiency
For researchers in neural network learning, this work challenges the default use of backpropagation by providing theoretical conditions where synthetic gradients offer better sample efficiency.
This paper theoretically analyzes when synthetic gradients can outperform backpropagation in terms of sample efficiency, showing that synthetic gradients can achieve arbitrarily lower gradient-estimation mean squared error under certain conditions, with experiments on contextual bandits and reinforcement learning tasks.
Backpropagation is the default learning rule for artificial neural networks and is often treated as the settled approach whenever differentiability is available. In this work, we revisit this convention through a theoretical lens of sample efficiency. We introduce a unified vectorized feedback framework for loss-based and reward-based learning on computational graphs, in which synthetic gradients emerge as a natural alternative to backpropagation. We characterize the conditions under which synthetic gradients can achieve a lower gradient-estimation mean squared error than backpropagation. We construct examples illustrating that this sample efficiency advantage can be arbitrarily large. Experiments on contextual bandits and reinforcement learning tasks demonstrate the potential of our theoretical findings.