Python Wrapper for Simulating Multi-Fidelity Optimization on HPO Benchmarks without Any Wait
This is an incremental tool for researchers and practitioners in machine learning to speed up multi-fidelity optimization simulations on HPO benchmarks.
The paper tackles the inefficiency of simulating asynchronous hyperparameter optimization (HPO) by introducing a Python wrapper that forces workers to wait for the actual runtime, reducing wait times from hours to 0.01 seconds while maintaining the same evaluation order as real experiments.
Hyperparameter (HP) optimization of deep learning (DL) is essential for high performance. As DL often requires several hours to days for its training, HP optimization (HPO) of DL is often prohibitively expensive. This boosted the emergence of tabular or surrogate benchmarks, which enable querying the (predictive) performance of DL with a specific HP configuration in a fraction. However, since the actual runtime of a DL training is significantly different from its query response time, simulators of an asynchronous HPO, e.g. multi-fidelity optimization, must wait for the actual runtime at each iteration in a naïve implementation; otherwise, the evaluation order during simulation does not match with the real experiment. To ease this issue, we developed a Python wrapper and describe its usage. This wrapper forces each worker to wait so that we yield exactly the same evaluation order as in the real experiment with only $10^{-2}$ seconds of waiting instead of waiting several hours. Our implementation is available at https://github.com/nabenabe0928/mfhpo-simulator/.