Poisson Process for Bayesian Optimization
This work addresses the need for more efficient and noise-robust optimization methods in machine learning, particularly for tasks like hyperparameter tuning, though it is incremental as it builds on existing ranking-based approaches.
The authors tackled the problem of Bayesian optimization by proposing a ranking-based surrogate model using a Poisson process, which reduces computational costs and improves robustness to noise compared to Gaussian process-based methods, as demonstrated in experiments on hyperparameter optimization and neural architecture search benchmarks.
BayesianOptimization(BO) is a sample-efficient black-box optimizer, and extensive methods have been proposed to build the absolute function response of the black-box function through a probabilistic surrogate model, including Tree-structured Parzen Estimator (TPE), random forest (SMAC), and Gaussian process (GP). However, few methods have been explored to estimate the relative rankings of candidates, which can be more robust to noise and have better practicality than absolute function responses, especially when the function responses are intractable but preferences can be acquired. To this end, we propose a novel ranking-based surrogate model based on the Poisson process and introduce an efficient BO framework, namely Poisson Process Bayesian Optimization (PoPBO). Two tailored acquisition functions are further derived from classic LCB and EI to accommodate it. Compared to the classic GP-BO method, our PoPBO has lower computation costs and better robustness to noise, which is verified by abundant experiments. The results on both simulated and real-world benchmarks, including hyperparameter optimization (HPO) and neural architecture search (NAS), show the effectiveness of PoPBO.