Can Large Language Models Find And Fix Vulnerable Software?
This addresses the problem of improving software security for developers and organizations by demonstrating LLMs' potential as a more effective tool, though it is incremental as it builds on existing LLM capabilities.
The study evaluated GPT-4's ability to detect and fix software vulnerabilities, finding it identified about four times more vulnerabilities than traditional analyzers and reduced vulnerabilities by 90% with only an 11% increase in code lines.
In this study, we evaluated the capability of Large Language Models (LLMs), particularly OpenAI's GPT-4, in detecting software vulnerabilities, comparing their performance against traditional static code analyzers like Snyk and Fortify. Our analysis covered numerous repositories, including those from NASA and the Department of Defense. GPT-4 identified approximately four times the vulnerabilities than its counterparts. Furthermore, it provided viable fixes for each vulnerability, demonstrating a low rate of false positives. Our tests encompassed 129 code samples across eight programming languages, revealing the highest vulnerabilities in PHP and JavaScript. GPT-4's code corrections led to a 90% reduction in vulnerabilities, requiring only an 11% increase in code lines. A critical insight was LLMs' ability to self-audit, suggesting fixes for their identified vulnerabilities and underscoring their precision. Future research should explore system-level vulnerabilities and integrate multiple static code analyzers for a holistic perspective on LLMs' potential.