Random Feedback Alignment Algorithms to train Neural Networks: Why do they Align?
This addresses a theoretical gap in understanding alternative training methods for neural networks, though it is incremental as it builds on existing feedback alignment concepts.
The paper investigates why random feedback alignment algorithms, which replace some gradient terms with random values, still enable neural network training by analyzing the mechanism of gradient alignment. They find that alignment is a stability criterion for fixed points but not sufficient for good performance, and high alignment can sometimes lead to poor results.
Feedback alignment algorithms are an alternative to backpropagation to train neural networks, whereby some of the partial derivatives that are required to compute the gradient are replaced by random terms. This essentially transforms the update rule into a random walk in weight space. Surprisingly, learning still works with those algorithms, including training of deep neural networks. This is generally attributed to an alignment of the update of the random walker with the true gradient - the eponymous gradient alignment -- which drives an approximate gradient descend. The mechanism that leads to this alignment remains unclear, however. In this paper, we use mathematical reasoning and simulations to investigate gradient alignment. We observe that the feedback alignment update rule has fixed points, which correspond to extrema of the loss function. We show that gradient alignment is a stability criterion for those fixed points. It is only a necessary criterion for algorithm performance. Experimentally, we demonstrate that high levels of gradient alignment can lead to poor algorithm performance and that the alignment is not always driving the gradient descend.