Implant Global and Local Hierarchy Information to Sequence based Code Representation Models
This work addresses the limitation of existing sequence-based models in capturing complete hierarchical information in source code, which is crucial for tasks like variable scope detection, classification, and generation in software engineering.
The paper tackled the problem of learning hierarchical structure in source code representation by introducing hierarchical embeddings for global and local levels, and proposed the Hierarchy Transformer (HiT) model. The result showed that HiT outperformed state-of-the-art baselines on three tasks across eight datasets, demonstrating stable training efficiency.
Source code representation with deep learning techniques is an important research field. There have been many studies that learn sequential or structural information for code representation. But sequence-based models and non-sequence-models both have their limitations. Researchers attempt to incorporate structural information to sequence-based models, but they only mine part of token-level hierarchical structure information. In this paper, we analyze how the complete hierarchical structure influences the tokens in code sequences and abstract this influence as a property of code tokens called hierarchical embedding. The hierarchical embedding is further divided into statement-level global hierarchy and token-level local hierarchy. Furthermore, we propose the Hierarchy Transformer (HiT), a simple but effective sequence model to incorporate the complete hierarchical embeddings of source code into a Transformer model. We demonstrate the effectiveness of hierarchical embedding on learning code structure with an experiment on variable scope detection task. Further evaluation shows that HiT outperforms SOTA baseline models and show stable training efficiency on three source code-related tasks involving classification and generation tasks across 8 different datasets.