CMA-ES for Hyperparameter Optimization of Deep Neural Networks
This is an incremental improvement for researchers and practitioners in machine learning seeking efficient hyperparameter tuning methods.
The paper tackled hyperparameter optimization for deep neural networks by proposing CMA-ES as an alternative to methods like grid search and Bayesian optimization, demonstrating its performance in a toy example on MNIST with 30 GPUs.
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known for its state-of-the-art performance in derivative-free optimization. CMA-ES has some useful invariance properties and is friendly to parallel evaluations of solutions. We provide a toy example comparing CMA-ES and state-of-the-art Bayesian optimization algorithms for tuning the hyperparameters of a convolutional neural network for the MNIST dataset on 30 GPUs in parallel.