Software Vulnerability and Functionality Assessment using LLMs
This work addresses the tedious and expensive process of code review for software developers, though it is incremental as it applies existing prompting methods to new tasks.
The paper tackled the problem of automating code review by using Large Language Models (LLMs) to flag security vulnerabilities and validate software functionality, finding that proprietary models significantly outperformed open-source ones and that 36.7% of LLM-generated vulnerability descriptions matched true CWE vulnerabilities.
While code review is central to the software development process, it can be tedious and expensive to carry out. In this paper, we investigate whether and how Large Language Models (LLMs) can aid with code reviews. Our investigation focuses on two tasks that we argue are fundamental to good reviews: (i) flagging code with security vulnerabilities and (ii) performing software functionality validation, i.e., ensuring that code meets its intended functionality. To test performance on both tasks, we use zero-shot and chain-of-thought prompting to obtain final ``approve or reject'' recommendations. As data, we employ seminal code generation datasets (HumanEval and MBPP) along with expert-written code snippets with security vulnerabilities from the Common Weakness Enumeration (CWE). Our experiments consider a mixture of three proprietary models from OpenAI and smaller open-source LLMs. We find that the former outperforms the latter by a large margin. Motivated by promising results, we finally ask our models to provide detailed descriptions of security vulnerabilities. Results show that 36.7% of LLM-generated descriptions can be associated with true CWE vulnerabilities.