Security of LLM-generated Code: A Comparative Analysis
For software developers and organizations using LLM-generated code, this work highlights significant security risks that need to be addressed.
The study evaluated code generated by seven popular LLMs and found that all models produce code containing vulnerabilities, most of which are critical or high severity.
The majority of software developers use or are planning to use Artificial Intelligence (AI) tools in their development processes. Their top reasons include improving productivity and faster learning. In fact, Large Language Model (LLM)-generated code is currently in production, including in major tech companies. However, concerns were raised about the risks associated with the use of AI tools to generate code. In this paper, we focus our attention on the risks to software security. We empirically evaluate the security of code generated by seven popular LLMs. We build upon previous work to mimic the behaviours of developers when using LLMs to generate code. Our results show that all seven LLMs that we have evaluated generate code that contains vulnerabilities, the majority of which are of critical or high severity.