Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces
This addresses a specific issue in hyperparameter optimization for machine learning practitioners, offering an incremental improvement over existing methods for conditional search spaces.
The paper tackled the problem of estimating hyperparameter importance in conditional search spaces, where hyperparameters depend on others, by proposing condPED-ANOVA, which provides meaningful importance estimates that reflect the conditional structure, unlike naive adaptations that yield misleading results.
We propose conditional PED-ANOVA (condPED-ANOVA), a principled framework for estimating hyperparameter importance (HPI) in conditional search spaces, where the presence or domain of a hyperparameter can depend on other hyperparameters. Although the original PED-ANOVA provides a fast and efficient way to estimate HPI within the top-performing regions of the search space, it assumes a fixed, unconditional search space and therefore cannot properly handle conditional hyperparameters. To address this, we introduce a conditional HPI for top-performing regions and derive a closed-form estimator that accurately reflects conditional activation and domain changes. Experiments show that naive adaptations of existing HPI estimators yield misleading or uninterpretable importance estimates in conditional settings, whereas condPED-ANOVA consistently provides meaningful importances that reflect the underlying conditional structure.