Natural Language to Code Generation in Interactive Data Science Notebooks
This work addresses the challenge of AI pair programming for data scientists, though it is incremental as it builds on existing code generation methods.
The authors tackled the problem of automatically generating code from natural language in data science notebooks by creating ARCADE, a benchmark of 1082 NL-to-code problems, and developed PaChiNCo, a 62B code language model that significantly outperforms existing models.
Computational notebooks, such as Jupyter notebooks, are interactive computing environments that are ubiquitous among data scientists to perform data wrangling and analytic tasks. To measure the performance of AI pair programmers that automatically synthesize programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1082 code generation problems using the pandas data analysis framework in data science notebooks. ARCADE features multiple rounds of NL-to-code problems from the same notebook. It requires a model to understand rich multi-modal contexts, such as existing notebook cells and their execution states as well as previous turns of interaction. To establish a strong baseline on this challenging task, we develop PaChiNCo, a 62B code language model (LM) for Python computational notebooks, which significantly outperforms public code LMs. Finally, we explore few-shot prompting strategies to elicit better code with step-by-step decomposition and NL explanation, showing the potential to improve the diversity and explainability of model predictions.