Interim Report on Human-Guided Adaptive Hyperparameter Optimization with Multi-Fidelity Sprints
This incremental work addresses hyperparameter tuning for NLP researchers and practitioners, offering a more efficient method for model optimization.
The paper tackles hyperparameter optimization for natural language models by using a phased approach with multi-fidelity sprints and human guidance, resulting in improved efficiency and performance on entity and relation extraction tasks, though specific numbers are not provided.
This case study applies a phased hyperparameter optimization process to compare multitask natural language model variants that utilize multiphase learning rate scheduling and optimizer parameter grouping. We employ short, Bayesian optimization sessions that leverage multi-fidelity, hyperparameter space pruning, progressive halving, and a degree of human guidance. We utilize the Optuna TPE sampler and Hyperband pruner, as well as the Scikit-Learn Gaussian process minimization. Initially, we use efficient low-fidelity sprints to prune the hyperparameter space. Subsequent sprints progressively increase their model fidelity and employ hyperband pruning for efficiency. A second aspect of our approach is using a meta-learner to tune threshold values to resolve classification probabilities during inference. We demonstrate our method on a collection of variants of the 2021 Joint Entity and Relation Extraction model proposed by Eberts and Ulges.