CORAL: COde RepresentAtion Learning with Weakly-Supervised Transformers for Analyzing Data Analysis
This work addresses the challenge of understanding data science processes and scientific code patterns for researchers and toolkit builders, though it is incremental in applying weakly supervised transformers to a specific domain.
The authors tackled the problem of analyzing large-scale scientific source code by proposing a weakly supervised transformer-based model that learns joint representations from code and comments, achieving a 38% increase in accuracy over expert heuristics on a new classification task for data analysis stages.
Large scale analysis of source code, and in particular scientific source code, holds the promise of better understanding the data science process, identifying analytical best practices, and providing insights to the builders of scientific toolkits. However, large corpora have remained unanalyzed in depth, as descriptive labels are absent and require expert domain knowledge to generate. We propose a novel weakly supervised transformer-based architecture for computing joint representations of code from both abstract syntax trees and surrounding natural language comments. We then evaluate the model on a new classification task for labeling computational notebook cells as stages in the data analysis process from data import to wrangling, exploration, modeling, and evaluation. We show that our model, leveraging only easily-available weak supervision, achieves a 38% increase in accuracy over expert-supplied heuristics and outperforms a suite of baselines. Our model enables us to examine a set of 118,000 Jupyter Notebooks to uncover common data analysis patterns. Focusing on notebooks with relationships to academic articles, we conduct the largest ever study of scientific code and find that notebook composition correlates with the citation count of corresponding papers.