Scalable Forward-Forward Algorithm
This addresses the need for more modular and memory-efficient training methods in large networks, though it appears incremental as it builds on existing Forward-Forward concepts.
The paper tackled the problem of eliminating backpropagation in neural networks by proposing a scalable Forward-Forward algorithm that trains layers separately, achieving performance comparable to standard backpropagation and outperforming it in hybrid designs with similar training speed.
We propose a scalable Forward-Forward (FF) algorithm that eliminates the need for backpropagation by training each layer separately. Unlike backpropagation, FF avoids backward gradients and can be more modular and memory efficient, making it appealing for large networks. We extend FF to modern convolutional architectures, such as MobileNetV3 and ResNet18, by introducing a new way to compute losses for convolutional layers. Experiments show that our method achieves performance comparable to standard backpropagation. Furthermore, when we divide the network into blocks, such as the residual blocks in ResNet, and apply backpropagation only within each block, but not across blocks, our hybrid design tends to outperform backpropagation baselines while maintaining a similar training speed. Finally, we present experiments on small datasets and transfer learning that confirm the adaptability of our method.