Improving Code Autocompletion with Transfer Learning
This addresses the challenge of training code autocompletion models with scarce data for programmers, though it is incremental as it builds on existing transfer learning methods.
The paper tackles the problem of limited IDE autocompletion training data by using unsupervised pretraining on non-IDE, non-autocompletion, and different-language code sequences, resulting in accuracy improvements of over 50% on small datasets and over 10% on 50k examples, with real-world A/B testing showing up to 6.63% increased autocompletion usage.
Software language models have achieved promising results predicting code completion usages, and several industry studies have described successful IDE integrations. Recently, accuracy in autocompletion prediction improved 12.8% from training on a real-world dataset collected from programmers' IDE activity. But what if limited examples of IDE autocompletion in the target programming language are available for model training? In this paper, we investigate the efficacy of pretraining autocompletion models on non-IDE, non-autocompletion, and different-language example code sequences. We find that these unsupervised pretrainings improve model accuracy by over 50% on very small fine-tuning datasets and over 10% on 50k labeled examples. We confirm the real-world impact of these pretrainings in an online setting through A/B testing on thousands of IDE autocompletion users, finding that pretraining is responsible for increases of up to 6.63% autocompletion usage.