Graph Conditioned Sparse-Attention for Improved Source Code Understanding
This addresses the problem of inefficient source code understanding for developers and AI systems by providing a more scalable model, though it is incremental as it builds on existing transformer and graph-based approaches.
The paper tackled the computational intractability of fusing graph representations with source code sequences in transformers by proposing a graph-conditioned sparse-attention mechanism, achieving state-of-the-art results in code summarization metrics and near state-of-the-art accuracy in variable misuse tasks, with linear memory and inference time growth compared to quadratic growth in prior methods.
Transformer architectures have been successfully used in learning source code representations. The fusion between a graph representation like Abstract Syntax Tree (AST) and a source code sequence makes the use of current approaches computationally intractable for large input sequence lengths. Source code can have long-range dependencies that require larger sequence lengths to model effectively. Current approaches have a quadratic growth in computational and memory costs with respect to the sequence length. Using such models in practical scenarios is difficult. In this work, we propose the conditioning of a source code snippet with its graph modality by using the graph adjacency matrix as an attention mask for a sparse self-attention mechanism and the use of a graph diffusion mechanism to model longer-range token dependencies. Our model reaches state-of-the-art results in BLEU, METEOR, and ROUGE-L metrics for the code summarization task and near state-of-the-art accuracy in the variable misuse task. The memory use and inference time of our model have linear growth with respect to the input sequence length as compared to the quadratic growth of previous works.