Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks
This work addresses the theoretical understanding of efficient alternatives to backpropagation for neural network training, offering insights into initialization effects, but it is incremental as it builds on existing Feedback Alignment research.
The paper provides theoretical convergence guarantees for the Feedback Alignment algorithm in deep linear networks and identifies that specific initializations can lead to implicit anti-regularization, where less important components are learned first, potentially harming effectiveness, while other initializations offer implicit regularization by learning components in order of importance.
We theoretically analyze the Feedback Alignment (FA) algorithm, an efficient alternative to backpropagation for training neural networks. We provide convergence guarantees with rates for deep linear networks for both continuous and discrete dynamics. Additionally, we study incremental learning phenomena for shallow linear networks. Interestingly, certain specific initializations imply that negligible components are learned before the principal ones, thus potentially negatively affecting the effectiveness of such a learning algorithm; a phenomenon we classify as implicit anti-regularization. We also provide initialization schemes where the components of the problem are approximately learned by decreasing order of importance, thus providing a form of implicit regularization.