LLM vs. SAST: A Technical Analysis on Detecting Coding Bugs of GPT4-Advanced Data Analysis
For software developers and security teams, this provides evidence that LLMs can outperform traditional SAST tools in detecting coding bugs, though the study is limited to a small curated dataset.
The study compares GPT-4 Advanced Data Analysis against two SAST tools (SonarQube and Cloud Defence) on 32 security scenarios, finding GPT-4 detects 93.75% of bugs versus 34.38% for SAST, with statistically significant superiority.
Large language models (LLMs) are increasingly used for code understanding, yet their practical effectiveness for vulnerability detection relative to Static Application Security Testing (SAST) remains insufficiently quantified. We present a controlled comparative study between GPT-4 (Advanced Data Analysis) and two SAST tools (SonarQube and Cloud Defence) on 32 curated security scenarios representing common coding pitfalls. Each scenario is scored with a binary detection rule, the two SAST outputs are aggregated using a logical OR baseline, and paired outcomes are evaluated using McNemar's test for statistical significance. In our dataset, GPT-4 correctly detected 30 of 32 scenarios (93.75\%), while the aggregated SAST baseline detected 11 of 32. The paired comparison shows a statistically significant difference in detection performance in favour of GPT-4. We also discuss security considerations and operational constraints for integrating LLM-enhanced vulnerability scanning into secure software development workflows.