Unsupervised Learning of General-Purpose Embeddings for Code Changes
This addresses the need for general-purpose embeddings in software engineering tasks like code modification and documentation, though it appears incremental as it builds on existing unsupervised and supervised techniques.
The paper tackles the problem of creating numerical representations for code changes by proposing an unsupervised pre-training approach that learns to correctly apply code changes. The model outperformed baselines by 5.9 percentage points in accuracy on code change application and matched supervised models on commit message generation.
Applying machine learning to tasks that operate with code changes requires their numerical representation. In this work, we propose an approach for obtaining such representations during pre-training and evaluate them on two different downstream tasks - applying changes to code and commit message generation. During pre-training, the model learns to apply the given code change in a correct way. This task requires only code changes themselves, which makes it unsupervised. In the task of applying code changes, our model outperforms baseline models by 5.9 percentage points in accuracy. As for the commit message generation, our model demonstrated the same results as supervised models trained for this specific task, which indicates that it can encode code changes well and can be improved in the future by pre-training on a larger dataset of easily gathered code changes.