Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
For deep learning researchers, this work challenges the necessity of gradient-based training by showing a gradient-free method can train deep networks, though it is currently limited to small-scale tasks.
This paper demonstrates that a simple Monte Carlo algorithm, which randomly mutates parameters and accepts changes only if loss decreases, can train deep neural networks without backpropagation, achieving feasibility on networks with over 20 layers and up to 16,384 hidden neurons, as well as a simple Transformer on MNIST and Tiny Shakespeare.
Backpropagation (BP) dominates deep learning training, but its reliance on gradients brings inherent troubles -- vanishing and exploding gradients. The pursuit of gradient-free methods has long been a goal in the field of artificial intelligence. This paper shows that indeed the simplest Monte Carlo algorithm implemented on a single GPU -- randomly mutate a parameter, keep it if the loss decreases, otherwise retry -- can practically train deep networks. This gradient-free method does not even need common techniques such as batch normalization or residual connections to directly train sufficiently deep networks. More remarkably, its flexibility extends to several nontrivial scenarios: it enables pure pruning training, supports discrete weights, accommodates unconventional transfer functions such as Gaussian, and reveals the substantial redundancy of deep networks. We have demonstrated its feasibility on deep networks with more than 20 layers, single-hidden-layer wide networks with up to 16,384 hidden neurons, and even a simple Transformer architecture trained on both image classification (MNIST) and character-level language modeling (Tiny Shakespeare). This simple gradient-free method may offer a complementary perspective for understanding the self-organization and learning mechanisms of neural networks, and also provides an alternative route for building physically inspired deep learning systems.