Diverse LLMs vs. Vulnerabilities: Who Detects and Fixes Them Better?
This addresses the challenge of enhancing security in software development by reducing error rates in vulnerability detection and repair, though it involves trade-offs that require careful tuning.
The study tackled the problem of improving software vulnerability detection and repair by aggregating outputs from diverse large language models, achieving 10-12% higher detection accuracy and significant recall and F1 score improvements for multi-file vulnerabilities.
Large Language Models (LLMs) are increasingly being studied for Software Vulnerability Detection (SVD) and Repair (SVR). Individual LLMs have demonstrated code understanding abilities, but they frequently struggle when identifying complex vulnerabilities and generating fixes. This study presents DVDR-LLM, an ensemble framework that combines outputs from diverse LLMs to determine whether aggregating multiple models reduces error rates. Our evaluation reveals that DVDR-LLM achieves 10-12% higher detection accuracy compared to the average performance of individual models, with benefits increasing as code complexity grows. For multi-file vulnerabilities, the ensemble approach demonstrates significant improvements in recall (+18%) and F1 score (+11.8%) over individual models. However, the approach raises measurable trade-offs: reducing false positives in verification tasks while simultaneously increasing false negatives in detection tasks, requiring careful decision on the required level of agreement among the LLMs (threshold) for increased performance across different security contexts. Artifact: https://github.com/Erroristotle/DVDR_LLM