Optimising Optimisers with Push GP
This work addresses the challenge of discovering novel optimization strategies in an automated way, which could benefit researchers and practitioners in optimization and machine learning, though it appears incremental as it builds on existing genetic programming techniques.
The paper tackled the problem of automatically designing optimizers for continuous-valued problems using Push GP, and found that the evolved optimizers generalized well to larger and unseen problems, sometimes outperforming existing methods like CMA-ES.
This work uses Push GP to automatically design both local and population-based optimisers for continuous-valued problems. The optimisers are trained on a single function optimisation landscape, using random transformations to discourage overfitting. They are then tested for generality on larger versions of the same problem, and on other continuous-valued problems. In most cases, the optimisers generalise well to the larger problems. Surprisingly, some of them also generalise very well to previously unseen problems, outperforming existing general purpose optimisers such as CMA-ES. Analysis of the behaviour of the evolved optimisers indicates a range of interesting optimisation strategies that are not found within conventional optimisers, suggesting that this approach could be useful for discovering novel and effective forms of optimisation in an automated manner.