Natural Language-Focused Software Engineering via Code-Documentation Equivalence
For software engineers and LLM-based tools, this work addresses the problem of inaccurate documentation, but the approach is incremental as it builds on existing LLM capabilities.
The paper introduces documentation-to-code equivalence, a property ensuring documentation accurately describes code, and presents Documentary, an approach that generates equivalent documentation for 53.4% of function-level code snippets. Using this equivalence, an LLM predicts function outputs with 12.8–24.5% higher accuracy compared to human-written or baseline-generated documentation.
Source code documentation is an integral part of software development and maintenance, as it helps in understanding the code and facilitates communication among developers. However, existing documentation is often incomplete, outdated, or inaccurate, which can lead to misunderstandings and errors. In the era of large language models (LLMs), which are being extensively used for software engineering tasks, the quality of documentation becomes even more critical, as documentation provides important context for the models. In this paper, we introduce the notion of documentation-to-code equivalence, a novel property that captures whether documentation accurately and completely describes the code it documents. We present a novel approach, called Documentary, to automatically generate equivalent documentation for a given code snippet. Our evaluation shows that Documentary can generate equivalent documentation for 53.4% of the evaluated function-level code snippets. To show the benefits of documentation-to-code equivalence, we describe and evaluate two software engineering tasks: code understanding and code editing. Our results show that documentation-to-code equivalence allows an LLM to predict the output of a function with 12.8--24.5% higher accuracy, when compared to human-written documentation and documentation generated by a baseline approach. Furthermore, human developers consider documentation generated by Documentary to be more useful for understanding and editing code than the original human-written documentation.