Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance
This work addresses the need for better empirical performance in parameter tuning for researchers and practitioners using TPE in frameworks like Hyperopt and Optuna, but it is incremental as it focuses on optimizing an existing method.
The paper tackles the problem of understanding the roles of control parameters in the Tree-structured Parzen Estimator (TPE) for Bayesian optimization, and through ablation studies on diverse benchmarks, it demonstrates that a recommended setting improves TPE's performance.
Recent scientific advances require complex experiment design, necessitating the meticulous tuning of many experiment parameters. Tree-structured Parzen estimator (TPE) is a widely used Bayesian optimization method in recent parameter tuning frameworks such as Hyperopt and Optuna. Despite its popularity, the roles of each control parameter in TPE and the algorithm intuition have not been discussed so far. The goal of this paper is to identify the roles of each control parameter and their impacts on parameter tuning based on the ablation studies using diverse benchmark datasets. The recommended setting concluded from the ablation studies is demonstrated to improve the performance of TPE. Our TPE implementation used in this paper is available at https://github.com/nabenabe0928/tpe/tree/single-opt.