Seamlessly Integrating Tree-Based Positional Embeddings into Transformer Models for Source Code Representation
This addresses the problem of inadequate source code representation for developers and researchers in software engineering, offering an incremental improvement by enhancing existing transformer models with explicit structural information.
The paper tackles the problem that traditional positional embeddings in transformer models fail to capture the hierarchical structure of source code represented as Abstract Syntax Trees (ASTs), and proposes a novel tree-based positional embedding approach that integrates hierarchical relationships like node depth and sibling indices into CodeBERTa, resulting in consistent improvements over the baseline in masked language modeling and clone detection tasks across metrics like loss, accuracy, F1 score, precision, and recall.
Transformer-based models have demonstrated significant success in various source code representation tasks. Nonetheless, traditional positional embeddings employed by these models inadequately capture the hierarchical structure intrinsic to source code, typically represented as Abstract Syntax Trees (ASTs). To address this, we propose a novel tree-based positional embedding approach that explicitly encodes hierarchical relationships derived from ASTs, including node depth and sibling indices. These hierarchical embeddings are integrated into the transformer architecture, specifically enhancing the CodeBERTa model. We thoroughly evaluate our proposed model through masked language modeling (MLM) pretraining and clone detection fine-tuning tasks. Experimental results indicate that our Tree-Enhanced CodeBERTa consistently surpasses the baseline model in terms of loss, accuracy, F1 score, precision, and recall, emphasizing the importance of incorporating explicit structural information into transformer-based representations of source code.