NeuroEvoBench: Benchmarking Evolutionary Optimizers for Deep Learning Applications
This work addresses the challenge for newcomers to hardware-accelerated EO and aims to facilitate broader adoption in deep learning by providing practical insights and benchmarks.
The authors tackled the lack of hyperparameter understanding and best practices for evolutionary optimization (EO) in deep learning by establishing NeuroEvoBench, a new benchmark tailored for deep learning applications, and exhaustively evaluated traditional and meta-learned EO methods, investigating core scientific questions like resource allocation and scalability.
Recently, the Deep Learning community has become interested in evolutionary optimization (EO) as a means to address hard optimization problems, e.g. meta-learning through long inner loop unrolls or optimizing non-differentiable operators. One core reason for this trend has been the recent innovation in hardware acceleration and compatible software - making distributed population evaluations much easier than before. Unlike for gradient descent-based methods though, there is a lack of hyperparameter understanding and best practices for EO - arguably due to severely less 'graduate student descent' and benchmarking being performed for EO methods. Additionally, classical benchmarks from the evolutionary community provide few practical insights for Deep Learning applications. This poses challenges for newcomers to hardware-accelerated EO and hinders significant adoption. Hence, we establish a new benchmark of EO methods (NeuroEvoBench) tailored toward Deep Learning applications and exhaustively evaluate traditional and meta-learned EO. We investigate core scientific questions including resource allocation, fitness shaping, normalization, regularization & scalability of EO. The benchmark is open-sourced at https://github.com/neuroevobench/neuroevobench under Apache-2.0 license.