DREAM: Debugging and Repairing AutoML Pipelines
This addresses reliability issues in AutoML for developers and users, but it is incremental as it builds on existing AutoML workflows.
The paper tackles bugs in AutoML systems, specifically performance and ineffective search bugs, by introducing DREAM, an automatic debugging and repairing system that expands search space and uses feedback-driven search, resulting in effective and efficient bug repair as shown in evaluation.
Deep Learning models have become an integrated component of modern software systems. In response to the challenge of model design, researchers proposed Automated Machine Learning (AutoML) systems, which automatically search for model architecture and hyperparameters for a given task. Like other software systems, existing AutoML systems suffer from bugs. We identify two common and severe bugs in AutoML, performance bug (i.e., searching for the desired model takes an unreasonably long time) and ineffective search bug (i.e., AutoML systems are not able to find an accurate enough model). After analyzing the workflow of AutoML, we observe that existing AutoML systems overlook potential opportunities in search space, search method, and search feedback, which results in performance and ineffective search bugs. Based on our analysis, we design and implement DREAM, an automatic debugging and repairing system for AutoML systems. It monitors the process of AutoML to collect detailed feedback and automatically repairs bugs by expanding search space and leveraging a feedback-driven search strategy. Our evaluation results show that DREAM can effectively and efficiently repair AutoML bugs.