Principled Training of Neural Networks with Direct Feedback Alignment
This work addresses the challenge of making biologically plausible training methods scalable for neural networks, though it appears incremental as it builds on existing feedback alignment techniques.
The paper tackled the problem of scaling direct feedback alignment methods beyond simple tasks like MNIST or CIFAR-10, which have failed due to a lack of standards and a bottleneck effect in narrow layers, and presented best practices based on alignment angle observations to address this.
The backpropagation algorithm has long been the canonical training method for neural networks. Modern paradigms are implicitly optimized for it, and numerous guidelines exist to ensure its proper use. Recently, synthetic gradients methods -where the error gradient is only roughly approximated - have garnered interest. These methods not only better portray how biological brains are learning, but also open new computational possibilities, such as updating layers asynchronously. Even so, they have failed to scale past simple tasks like MNIST or CIFAR-10. This is in part due to a lack of standards, leading to ill-suited models and practices forbidding such methods from performing to the best of their abilities. In this work, we focus on direct feedback alignment and present a set of best practices justified by observations of the alignment angles. We characterize a bottleneck effect that prevents alignment in narrow layers, and hypothesize it may explain why feedback alignment methods have yet to scale to large convolutional networks.