Automating Data Science: Prospects and Challenges
This work addresses the problem of automating data science processes for data scientists, but it is incremental as it reviews existing prospects and challenges without introducing new methods.
The paper examines the potential of automation to transform data science by facilitating rather than replacing human experts, noting that while techniques like AutoML are automating modeling stages, open-ended and context-dependent tasks remain challenging to automate.
Given the complexity of typical data science projects and the associated demand for human expertise, automation has the potential to transform the data science process. Key insights: * Automation in data science aims to facilitate and transform the work of data scientists, not to replace them. * Important parts of data science are already being automated, especially in the modeling stages, where techniques such as automated machine learning (AutoML) are gaining traction. * Other aspects are harder to automate, not only because of technological challenges, but because open-ended and context-dependent tasks require human interaction.