AutoML: Exploration v.s. Exploitation
This work addresses the time and resource efficiency of AutoML systems for practitioners, but the findings are incremental as they confirm existing exploration-exploitation trade-offs without new methods.
The paper empirically tests whether focusing on the most promising classifiers improves AutoML pipeline performance under time constraints, finding that exploiting top classifiers does not yield statistically better results than exploring the entire search space, even with longer time budgets.
Building a machine learning (ML) pipeline in an automated way is a crucial and complex task as it is constrained with the available time budget and resources. This encouraged the research community to introduce several solutions to utilize the available time and resources. A lot of work is done to suggest the most promising classifiers for a given dataset using sundry of techniques including meta-learning based techniques. This gives the autoML framework the chance to spend more time exploiting those classifiers and tuning their hyper-parameters. In this paper, we empirically study the hypothesis of improving the pipeline performance by exploiting the most promising classifiers within the limited time budget. We also study the effect of increasing the time budget over the pipeline performance. The empirical results across autoSKLearn, TPOT and ATM, show that exploiting the most promising classifiers does not achieve a statistically better performance than exploring the entire search space. The same conclusion is also applied for long time budgets.