PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces
This work addresses the need for efficient hyperparameter importance analysis in machine learning, particularly for algorithm designers, though it is incremental as it builds on existing f-ANOVA methods.
The paper tackles the problem of quantifying hyperparameter importance in arbitrary subspaces, such as those defined by top performance, by deriving a novel formulation of functional ANOVA and proposing an algorithm called PED-ANOVA that uses Pearson divergence for closed-form calculation, resulting in successful identification of important hyperparameters with high computational efficiency.
The recent rise in popularity of Hyperparameter Optimization (HPO) for deep learning has highlighted the role that good hyperparameter (HP) space design can play in training strong models. In turn, designing a good HP space is critically dependent on understanding the role of different HPs. This motivates research on HP Importance (HPI), e.g., with the popular method of functional ANOVA (f-ANOVA). However, the original f-ANOVA formulation is inapplicable to the subspaces most relevant to algorithm designers, such as those defined by top performance. To overcome this issue, we derive a novel formulation of f-ANOVA for arbitrary subspaces and propose an algorithm that uses Pearson divergence (PED) to enable a closed-form calculation of HPI. We demonstrate that this new algorithm, dubbed PED-ANOVA, is able to successfully identify important HPs in different subspaces while also being extremely computationally efficient.