A Comparative Analysis of Large Language Models for Code Documentation Generation
This work addresses the need for efficient and high-quality code documentation generation for software developers, but it is incremental as it primarily compares existing models without introducing new methods.
This paper tackled the problem of generating code documentation by comparing the performance of several large language models, finding that closed-source models like GPT-3.5, GPT-4, and Bard consistently outperformed open-source alternatives and original documentation across parameters such as accuracy and completeness, while file-level documentation performed worse than inline and function-level documentation.
This paper presents a comprehensive comparative analysis of Large Language Models (LLMs) for generation of code documentation. Code documentation is an essential part of the software writing process. The paper evaluates models such as GPT-3.5, GPT-4, Bard, Llama2, and Starchat on various parameters like Accuracy, Completeness, Relevance, Understandability, Readability and Time Taken for different levels of code documentation. Our evaluation employs a checklist-based system to minimize subjectivity, providing a more objective assessment. We find that, barring Starchat, all LLMs consistently outperform the original documentation. Notably, closed-source models GPT-3.5, GPT-4, and Bard exhibit superior performance across various parameters compared to open-source/source-available LLMs, namely LLama 2 and StarChat. Considering the time taken for generation, GPT-4 demonstrated the longest duration, followed by Llama2, Bard, with ChatGPT and Starchat having comparable generation times. Additionally, file level documentation had a considerably worse performance across all parameters (except for time taken) as compared to inline and function level documentation.