Ranindya Paramitha

Imranur Rahman, Jill Marley, Ranindya Paramitha et al.

Dependency resolution at a specified point in time in the past can provide insight into software evolution in software ecosystems and facilitate the design of dynamic metrics (e.g., dependency freshness, dependency update rhythm). However, dependency resolution at specified points in time is not possible in major software ecosystems due to a lack of support from package management tools. The goal of this paper is to aid practitioners and researchers in analyzing the state of the ecosystem dependency graph at release points using an enriched dataset with dependency resolution at release points for npm, PyPI, and crates.io packages. In this paper, we present a methodology to construct dependency resolution at release points of packages in software ecosystems, which we enrich with vulnerability data from OSV. We apply our methodology to construct CHRONO-RESOLUTION, a dataset of dependency resolution at release points for npm, PyPI, and crates.io packages, and release it for future research.

Ranindya Paramitha, Yuan Feng, Fabio Massacci

Vulnerability datasets used for ML testing implicitly contain retrospective information. When tested on the field, one can only use the labels available at the time of training and testing (e.g. seen and assumed negatives). As vulnerabilities are discovered across calendar time, labels change and past performance is not necessarily aligned with future performance. Past works only considered the slices of the whole history (e.g. DiverseVUl) or individual differences between releases (e.g. Jimenez et al. ESEC/FSE 2019). Such approaches are either too optimistic in training (e.g. the whole history) or too conservative (e.g. consecutive releases). We propose a method to restructure a dataset into a series of datasets in which both training and testing labels change to account for the knowledge available at the time. If the model is actually learning, it should improve its performance over time as more data becomes available and data becomes more stable, an effect that can be checked with the Mann-Kendall test. We validate our methodology for vulnerability detection with 4 time-based datasets (3 projects from BigVul dataset + Vuldeepecker's NVD) and 5 ML models (Code2Vec, CodeBERT, LineVul, ReGVD, and Vuldeepecker). In contrast to the intuitive expectation (more retrospective information, better performance), the trend results show that performance changes inconsistently across the years, showing that most models are not learning.