Closed-Form Feedback-Free Learning with Forward Projection
This work addresses the challenge of efficient and interpretable training for neural networks in settings where feedback is unavailable, offering a novel approach that could benefit applications in biomedical data analysis and beyond.
The paper tackles the problem of training neural networks without retrograde communication by proposing Forward Projection (FP), a closed-form method that uses a single forward pass over the dataset. It demonstrates that FP achieves generalization comparable to gradient-based methods in large-sample tasks and yields more generalizable models in some few-shot learning scenarios, with significant speed improvements.
State-of-the-art methods for backpropagation-free learning employ local error feedback to direct iterative optimisation via gradient descent. In this study, we examine the more restrictive setting where retrograde communication from neuronal outputs is unavailable for pre-synaptic weight optimisation. To address this challenge, we propose Forward Projection (FP). This randomised closed-form training method requires only a single forward pass over the entire dataset for model fitting, without retrograde communication. Our method generates target values for pre-activation membrane potentials at each layer through randomised nonlinear projections of pre-synaptic inputs and the labels, thereby encoding information from both sources. Local loss functions are optimised over pre-synaptic inputs using closed-form regression, without feedback from neuronal outputs or downstream layers. Interpretability is a key advantage of FP training; membrane potentials of hidden neurons in FP-trained networks encode information which are interpretable layer-wise as label predictions. We demonstrate the effectiveness of FP across four biomedical datasets, comparing it with backpropagation and local learning techniques such as Forward-Forward training and Local Supervision in multi-layer perceptron and convolutional architectures. In some few-shot learning tasks, FP yielded more generalisable models than those optimised via backpropagation. In large-sample tasks, FP-based models achieve generalisation comparable to gradient descent-based local learning methods while requiring only a single forward propagation step, achieving significant speed up for training.