Leader-Follower Neural Networks with Local Error Signals Inspired by Complex Collectives
This work addresses the problem of scalable and efficient training for neural networks, particularly in resource-limited settings, by offering a novel BP-free approach that shows competitive performance, though it is incremental in the context of alternative training methods.
The paper tackled the challenge of training neural networks without backpropagation by proposing a leader-follower architecture inspired by collective behavior in nature, achieving significantly lower error rates than previous BP-free methods on MNIST and CIFAR-10 and outperforming them on ImageNet.
The collective behavior of a network with heterogeneous, resource-limited information processing units (e.g., group of fish, flock of birds, or network of neurons) demonstrates high self-organization and complexity. These emergent properties arise from simple interaction rules where certain individuals can exhibit leadership-like behavior and influence the collective activity of the group. Motivated by the intricacy of these collectives, we propose a neural network (NN) architecture inspired by the rules observed in nature's collective ensembles. This NN structure contains workers that encompass one or more information processing units (e.g., neurons, filters, layers, or blocks of layers). Workers are either leaders or followers, and we train a leader-follower neural network (LFNN) by leveraging local error signals and optionally incorporating backpropagation (BP) and global loss. We investigate worker behavior and evaluate LFNNs through extensive experimentation. Our LFNNs trained with local error signals achieve significantly lower error rates than previous BP-free algorithms on MNIST and CIFAR-10 and even surpass BP-enabled baselines. In the case of ImageNet, our LFNN-l demonstrates superior scalability and outperforms previous BP-free algorithms by a significant margin.