Unlocking Deep Learning: A BP-Free Approach for Parallel Block-Wise Training of Neural Networks
This addresses the problem of training inefficiencies and biological implausibility in deep learning for researchers and practitioners, though it is incremental as it builds on existing local error signal methods.
The paper tackles the limitations of backpropagation in deep learning by proposing a block-wise BP-free approach that uses local error signals to train neural network blocks in parallel, achieving performance improvements over end-to-end backpropagation and other block-wise methods on datasets like CIFAR-10 and Tiny-ImageNet.
Backpropagation (BP) has been a successful optimization technique for deep learning models. However, its limitations, such as backward- and update-locking, and its biological implausibility, hinder the concurrent updating of layers and do not mimic the local learning processes observed in the human brain. To address these issues, recent research has suggested using local error signals to asynchronously train network blocks. However, this approach often involves extensive trial-and-error iterations to determine the best configuration for local training. This includes decisions on how to decouple network blocks and which auxiliary networks to use for each block. In our work, we introduce a novel BP-free approach: a block-wise BP-free (BWBPF) neural network that leverages local error signals to optimize distinct sub-neural networks separately, where the global loss is only responsible for updating the output layer. The local error signals used in the BP-free model can be computed in parallel, enabling a potential speed-up in the weight update process through parallel implementation. Our experimental results consistently show that this approach can identify transferable decoupled architectures for VGG and ResNet variations, outperforming models trained with end-to-end backpropagation and other state-of-the-art block-wise learning techniques on datasets such as CIFAR-10 and Tiny-ImageNet. The code is released at https://github.com/Belis0811/BWBPF.