Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale
This addresses the problem of inefficient hyper-parameter tuning for machine learning practitioners, offering incremental improvements in speed and scalability.
The paper tackles the scalability bottleneck in hyper-parameter tuning systems by proposing Hyper-Tune, an efficient distributed framework with system optimizations like automatic resource allocation and multi-fidelity methods, achieving up to 11.2x speedup over state-of-the-art methods.
The ever-growing demand and complexity of machine learning are putting pressure on hyper-parameter tuning systems: while the evaluation cost of models continues to increase, the scalability of state-of-the-arts starts to become a crucial bottleneck. In this paper, inspired by our experience when deploying hyper-parameter tuning in a real-world application in production and the limitations of existing systems, we propose Hyper-Tune, an efficient and robust distributed hyper-parameter tuning framework. Compared with existing systems, Hyper-Tune highlights multiple system optimizations, including (1) automatic resource allocation, (2) asynchronous scheduling, and (3) multi-fidelity optimizer. We conduct extensive evaluations on benchmark datasets and a large-scale real-world dataset in production. Empirically, with the aid of these optimizations, Hyper-Tune outperforms competitive hyper-parameter tuning systems on a wide range of scenarios, including XGBoost, CNN, RNN, and some architectural hyper-parameters for neural networks. Compared with the state-of-the-art BOHB and A-BOHB, Hyper-Tune achieves up to 11.2x and 5.1x speedups, respectively.