A Ground-Truth-Based Evaluation of Vulnerability Detection Across Multiple Ecosystems
For security researchers and practitioners, this work provides a reproducible benchmark for evaluating vulnerability detection tools, though it is an incremental contribution as it applies existing methods to a new curated dataset.
The authors created a ground-truth dataset from the OSV database to evaluate vulnerability detection tools across multiple ecosystems, revealing systematic differences between detection systems and emphasizing the need for transparent dataset construction.
Automated vulnerability detection tools are widely used to identify security vulnerabilities in software dependencies. However, the evaluation of such tools remains challenging due to the heterogeneous structure of vulnerability data sources, inconsistent identifier schemes, and ambiguities in version range specifications. In this paper, we present an empirical evaluation of vulnerability detection across multiple software ecosystems using a curated ground-truth dataset derived from the Open Source Vulnerabilities (OSV) database. The dataset explicitly maps vulnerabilities to concrete package versions and enables a systematic comparison of detection results across different tools and services. Since vulnerability databases such as OSV are continuously updated, the dataset used in this study represents a snapshot of the vulnerability landscape at the time of the evaluation. To support reproducibility and future studies, we provide an open-source tool that automatically reconstructs the dataset from the current OSV database using the methodology described in this paper. Our evaluation highlights systematic differences between vulnerability detection systems and demonstrates the importance of transparent dataset construction for reproducible empirical security research.