Modeling Code: Is Text All You Need?
This addresses a key bottleneck in code modeling for developers and researchers, though it appears incremental as it builds on prior structured approaches.
The paper tackles the limitation of transformer-based code LLMs in reasoning about structured properties like control and data flow by introducing a novel approach that combines text-based modeling with structured forms, aiming to enhance generative capabilities and scale.
Code LLMs have become extremely popular recently for modeling source code across a variety of tasks, such as generation, translation, and summarization. However, transformer-based models are limited in their capabilities to reason through structured, analytical properties of code, such as control and data flow. Previous work has explored the modeling of these properties with structured data and graph neural networks. However, these approaches lack the generative capabilities and scale of modern LLMs. In this work, we introduce a novel approach to combine the strengths of modeling both code as text and more structured forms.