Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection
This work addresses the generalization issue in vulnerability detection for software security, but it is incremental as it builds on existing methods with improved data and model benchmarking.
The paper tackles the problem of AI-based software vulnerability detection failing to generalize to unseen codebases by investigating the impact of model architecture, parameter configuration, and training data quality, resulting in a 6.8% improvement in recall on the BigVul benchmark and enhanced performance on unseen projects.
AI-based solutions demonstrate remarkable results in identifying vulnerabilities in software, but research has consistently found that this performance does not generalize to unseen codebases. In this paper, we specifically investigate the impact of model architecture, parameter configuration, and quality of training data on the ability of these systems to generalize. For this purpose, we introduce VulGate, a high quality state of the art dataset that mitigates the shortcomings of prior datasets, by removing mislabeled and duplicate samples, updating new vulnerabilities, incorporating additional metadata, integrating hard samples, and including dedicated test sets. We undertake a series of experiments to demonstrate that improved dataset diversity and quality substantially enhances vulnerability detection. We also introduce and benchmark multiple encoder-only and decoder-only models. We find that encoder-based models outperform other models in terms of accuracy and generalization. Our model achieves \textbf{6.8\%} improvement in recall on the benchmark BigVul dataset and outperforms others on unseen projects, demonstrating enhanced generalizability. Our results highlight the role of data quality and model selection in the development of robust vulnerability detection systems. Our findings suggest a direction for future systems with high cross-project effectiveness.