Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loop
This research addresses the challenge of integrating AutoML with human oversight in enterprise data science, but it is incremental as it builds on existing human-in-the-loop concepts.
The study investigated how enterprises use AutoML systems and the role of human oversight, finding that data visualization often fails to balance speed and oversight, leading to a framework for automation levels based on expertise.
AutoML systems can speed up routine data science work and make machine learning available to those without expertise in statistics and computer science. These systems have gained traction in enterprise settings where pools of skilled data workers are limited. In this study, we conduct interviews with 29 individuals from organizations of different sizes to characterize how they currently use, or intend to use, AutoML systems in their data science work. Our investigation also captures how data visualization is used in conjunction with AutoML systems. Our findings identify three usage scenarios for AutoML that resulted in a framework summarizing the level of automation desired by data workers with different levels of expertise. We surfaced the tension between speed and human oversight and found that data visualization can do a poor job balancing the two. Our findings have implications for the design and implementation of human-in-the-loop visual analytics approaches.