Limited Evaluation Evolutionary Optimization of Large Neural Networks
This work addresses the computational bottleneck for applying EAs to large neural networks, but it is incremental as it shows limited practical gains over standard methods.
The paper tackled training neural networks with evolutionary algorithms (EAs) by implementing a GPU-based method for batch evaluation, but found it led to a significant accuracy trade-off, achieving 97.6% test accuracy on MNIST with 92k parameters compared to 98% with Adam.
Stochastic gradient descent is the most prevalent algorithm to train neural networks. However, other approaches such as evolutionary algorithms are also applicable to this task. Evolutionary algorithms bring unique trade-offs that are worth exploring, but computational demands have so far restricted exploration to small networks with few parameters. We implement an evolutionary algorithm that executes entirely on the GPU, which allows to efficiently batch-evaluate a whole population of networks. Within this framework, we explore the limited evaluation evolutionary algorithm for neural network training and find that its batch evaluation idea comes with a large accuracy trade-off. In further experiments, we explore crossover operators and find that unprincipled random uniform crossover performs extremely well. Finally, we train a network with 92k parameters on MNIST using an EA and achieve 97.6 % test accuracy compared to 98 % test accuracy on the same network trained with Adam. Code is available at https://github.com/jprellberg/gpuea.