Hyperparameter Optimization with Differentiable Metafeatures
This work provides a strong specific gain in hyperparameter optimization performance for machine learning practitioners by improving the efficiency of finding optimal hyperparameters.
This paper introduces Differentiable Metafeature-based Surrogate (DMFBS), a cross-dataset surrogate model that predicts hyperparameter response (validation loss) by integrating a differentiable metafeature extractor. DMFBS consistently outperforms existing HPO models by an average of 10% on three large meta-datasets.
Metafeatures, or dataset characteristics, have been shown to improve the performance of hyperparameter optimization (HPO). Conventionally, metafeatures are precomputed and used to measure the similarity between datasets, leading to a better initialization of HPO models. In this paper, we propose a cross dataset surrogate model called Differentiable Metafeature-based Surrogate (DMFBS), that predicts the hyperparameter response, i.e. validation loss, of a model trained on the dataset at hand. In contrast to existing models, DMFBS i) integrates a differentiable metafeature extractor and ii) is optimized using a novel multi-task loss, linking manifold regularization with a dataset similarity measure learned via an auxiliary dataset identification meta-task, effectively enforcing the response approximation for similar datasets to be similar. We compare DMFBS against several recent models for HPO on three large meta-datasets and show that it consistently outperforms all of them with an average 10% improvement. Finally, we provide an extensive ablation study that examines the different components of our approach.