Processing Large Datasets of Fined Grained Source Code Changes
This work addresses the data processing bottleneck for researchers studying large-scale code repositories, though it is incremental as it builds on an existing framework.
The authors tackled the challenge of processing large datasets of fine-grained AST-level source code changes by extending the CodeDistillery framework with data manipulation capabilities, enabling the processing of dozens of millions of changes to automate repository mining and streamline data acquisition.
In the era of Big Code, when researchers seek to study an increasingly large number of repositories to support their findings, the data processing stage may require manipulating millions and more of records. In this work we focus on studies involving fine-grained AST level source code changes. We present how we extended the CodeDistillery source code mining framework with data manipulation capabilities, aimed to alleviate the processing of large datasets of fine grained source code changes. The capabilities we have introduced allow researchers to highly automate their repository mining process and streamline the data acquisition and processing phases. These capabilities have been successfully used to conduct a number of studies, in the course of which dozens of millions of fine-grained source code changes have been processed.