EDAssistant: Supporting Exploratory Data Analysis in Computational Notebooks with In-Situ Code Search and Recommendation
This tool addresses the problem of inefficient external information retrieval for data scientists during EDA, though it is incremental as it builds on existing notebook environments and search techniques.
The paper tackles the challenge of supporting exploratory data analysis (EDA) for novices or data scientists lacking specific knowledge by introducing EDAssistant, a JupyterLab extension that provides in-situ code search and recommendation using machine learning models trained on a large corpus of EDA notebooks. A user study showed that EDAssistant was effective and useful, with participants appreciating its smooth and in-context support compared to using external search engines.
Using computational notebooks (e.g., Jupyter Notebook), data scientists rationalize their exploratory data analysis (EDA) based on their prior experience and external knowledge such as online examples. For novices or data scientists who lack specific knowledge about the dataset or problem to investigate, effectively obtaining and understanding the external information is critical to carry out EDA. This paper presents EDAssistant, a JupyterLab extension that supports EDA with in-situ search of example notebooks and recommendation of useful APIs, powered by novel interactive visualization of search results. The code search and recommendation are enabled by state-of-the-art machine learning models, trained on a large corpus of EDA notebooks collected online. A user study is conducted to investigate both EDAssistant and data scientists' current practice (i.e., using external search engines). The results demonstrate the effectiveness and usefulness of EDAssistant, and participants appreciated its smooth and in-context support of EDA. We also report several design implications regarding code recommendation tools.