Bringing Structure to Naturalness: On the Naturalness of ASTs
This work addresses the problem of understanding and leveraging code structure for predictability in software engineering, though it is incremental in testing an existing hypothesis.
The paper investigates the Structured Naturalness Hypothesis, which posits that structured representations of code like ASTs are statistically predictable, and finds that TreeLSTM models on ASTs are competitive with n-gram models for languages like Ruby but worse for Java or Python, while also achieving near state-of-the-art results in defect prediction without manual features.
Source code comes in different shapes and forms. Previous research has already shown code to be more predictable than natural language as well as highlighted its statistical predictability at the token level: source code can be natural. More recently, the structure of code -- control flow, syntax graphs, abstract syntax trees etc. -- has been successfully used to improve the state-of-the-art on numerous tasks: code suggestion, code summarisation, method naming etc. This body of work implicitly assumes that structured representations of code are similarly statistically predictable, i.e. that a structured view of code is also natural. We consider that this view should be made explicit and propose directly studying the Structured Naturalness Hypothesis. Beyond just naming existing research that assumes this hypothesis and formulating it, we also provide evidence in the case of trees: TreeLSTM models over ASTs for some languages, such as Ruby, are competitive with $n$-gram models while handling the syntax token issue highlighted by previous research 'for free'. For other languages, such as Java or Python, we find tree models to perform worse, suggesting that downstream task improvement is uncorrelated to the language modelling task. Further, we show how such naturalness signals can be employed for near state-of-the-art results on just-in-time defect prediction while forgoing manual feature engineering work.