What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code
This work addresses the problem of understanding why pre-trained code models work, which is important for researchers and practitioners in code intelligence, though it is incremental as it builds on existing models without introducing new methods.
The paper tackles the lack of interpretability in pre-trained language models for source code by conducting a structural analysis of models like CodeBERT and GraphCodeBERT, revealing that attention aligns with syntax structure and models can preserve and induce syntax trees in their representations.
Recently, many pre-trained language models for source code have been proposed to model the context of code and serve as a basis for downstream code intelligence tasks such as code completion, code search, and code summarization. These models leverage masked pre-training and Transformer and have achieved promising results. However, currently there is still little progress regarding interpretability of existing pre-trained code models. It is not clear why these models work and what feature correlations they can capture. In this paper, we conduct a thorough structural analysis aiming to provide an interpretation of pre-trained language models for source code (e.g., CodeBERT, and GraphCodeBERT) from three distinctive perspectives: (1) attention analysis, (2) probing on the word embedding, and (3) syntax tree induction. Through comprehensive analysis, this paper reveals several insightful findings that may inspire future studies: (1) Attention aligns strongly with the syntax structure of code. (2) Pre-training language models of code can preserve the syntax structure of code in the intermediate representations of each Transformer layer. (3) The pre-trained models of code have the ability of inducing syntax trees of code. Theses findings suggest that it may be helpful to incorporate the syntax structure of code into the process of pre-training for better code representations.