Evaluation System for a Bayesian Optimization Service
This work addresses the problem of ensuring robustness in a Bayesian optimization service for machine learning practitioners, but it is incremental as it focuses on testing methodology rather than new optimization techniques.
The authors tackled the need for reliable testing of a Bayesian optimization service by developing an evaluation framework to measure the impact of changes, enabling research engineers to make confident and quick modifications to the core optimization engine.
Bayesian optimization is an elegant solution to the hyperparameter optimization problem in machine learning. Building a reliable and robust Bayesian optimization service requires careful testing methodology and sound statistical analysis. In this talk we will outline our development of an evaluation framework to rigorously test and measure the impact of changes to the SigOpt optimization service. We present an overview of our evaluation system and discuss how this framework empowers our research engineers to confidently and quickly make changes to our core optimization engine