Signal Propagation: A Framework for Learning and Inference In a Forward Pass
This addresses the problem of structural and computational constraints in neural network training for researchers in machine learning, neuroscience, and hardware design, offering a more efficient and biologically plausible alternative to backpropagation.
The authors introduced signal propagation (sigprop), a learning framework that updates neural network parameters using only a forward pass, eliminating the need for backpropagation's feedback connectivity or backward passes. They demonstrated that sigprop is more efficient in time and memory than existing methods and applied it to train continuous-time and spiking neural networks with biologically and hardware-compatible updates.
We propose a new learning framework, signal propagation (sigprop), for propagating a learning signal and updating neural network parameters via a forward pass, as an alternative to backpropagation. In sigprop, there is only the forward path for inference and learning. So, there are no structural or computational constraints necessary for learning to take place, beyond the inference model itself, such as feedback connectivity, weight transport, or a backward pass, which exist under backpropagation based approaches. That is, sigprop enables global supervised learning with only a forward path. This is ideal for parallel training of layers or modules. In biology, this explains how neurons without feedback connections can still receive a global learning signal. In hardware, this provides an approach for global supervised learning without backward connectivity. Sigprop by construction has compatibility with models of learning in the brain and in hardware than backpropagation, including alternative approaches relaxing learning constraints. We also demonstrate that sigprop is more efficient in time and memory than they are. To further explain the behavior of sigprop, we provide evidence that sigprop provides useful learning signals in context to backpropagation. To further support relevance to biological and hardware learning, we use sigprop to train continuous time neural networks with Hebbian updates, and train spiking neural networks with only the voltage or with biologically and hardware compatible surrogate functions.