Language Modelling for Source Code with Transformer-XL
This work addresses the problem of efficient language modeling for source code, which is incremental as it applies existing models to a new domain.
The paper evaluated neural language models for source code, finding that Transformer-XL outperforms RNN-based models in capturing software naturalness with significantly lower computational cost.
It has been found that software, like natural language texts, exhibits "naturalness", which can be captured by statistical language models. In recent years, neural language models have been proposed to represent the naturalness of software through deep learning. In this paper, we conduct an experimental evaluation of state-of-the-art neural language models for source code, including RNN-based models and Transformer-XL based models. Through experiments on a large-scale Python code corpus, we find that the Transformer-XL model outperforms RNN-based models (including LSTM and GRU models) in capturing the naturalness of software, with far less computational cost.