MOFA: Modular Factorial Design for Hyperparameter Optimization
This work provides an efficient and effective hyperparameter optimization method for machine learning practitioners, offering an incremental improvement over existing techniques.
This paper introduces MOFA, a hyperparameter optimization (HPO) method that iteratively explores hyperparameter space using factorial design and exploits results with factorial analysis to fix less impactful hyperparameters. MOFA is shown to achieve better effectiveness and efficiency compared to state-of-the-art methods.
This paper presents a novel and lightweight hyperparameter optimization (HPO) method, MOdular FActorial Design (MOFA). MOFA pursues several rounds of HPO, where each round alternates between exploration of hyperparameter space by factorial design and exploitation of evaluation results by factorial analysis. Each round first explores the configuration space by constructing a low-discrepancy set of hyperparameters that cover this space well while de-correlating hyperparameters, and then exploits evaluation results through factorial analysis that determines which hyperparameters should be further explored and which should become fixed in the next round. We prove that the inference of MOFA achieves higher confidence than other sampling schemes. Each individual round is highly parallelizable and hence offers major improvements of efficiency compared to model-based methods. Empirical results show that MOFA achieves better effectiveness and efficiency compared with state-of-the-art methods.