Learning with augmented target information: An alternative theory of Feedback Alignment
This addresses the biological plausibility issues in neural network training for researchers in computational neuroscience and machine learning, though it is incremental as it builds on existing FA methods.
The paper tackles the lack of a satisfying explanation for how Feedback Alignment (FA) works across different neural network architectures by proposing a novel, architecture-agnostic theory based on information theory, showing that FA learns effective representations by embedding target information into networks, and demonstrates this through analysis and experiments, with designed variants achieving comparable performance on several tasks.
While error backpropagation (BP) has dominated the training of nearly all modern neural networks for a long time, it suffers from several biological plausibility issues such as the symmetric weight requirement and synchronous updates. Feedback Alignment (FA) was proposed as an alternative to BP to address those dilemmas and has been demonstrated to be effective on various tasks and network architectures. Despite its simplicity and effectiveness, a satisfying explanation of how FA works across different architectures is still lacking. Here we propose a novel, architecture-agnostic theory of how FA works through the lens of information theory: Instead of approximating gradients calculated by BP with the same parameter, FA learns effective representations by embedding target information into neural networks to be trained. We show this through the analysis of FA dynamics in idealized settings and then via a series of experiments. Based on the implications of this theory, we designed three variants of FA and show their comparable performance on several tasks. These variants also account for some phenomena and theories in neuroscience such as predictive coding and representational drift.