Learning-based Models for Vulnerability Detection: An Extensive Study
This work addresses the need for better understanding and practical application of learning-based vulnerability detection models, though it is incremental as it evaluates existing methods rather than introducing new ones.
The study investigated deep learning-based models for vulnerability detection, comparing sequence-based and graph-based approaches on a large-scale dataset, finding that sequence-based models outperform graph-based models and LLMs like ChatGPT, and revealing model instability with semantically equivalent input changes.
Though many deep learning-based models have made great progress in vulnerability detection, we have no good understanding of these models, which limits the further advancement of model capability, understanding of the mechanism of model detection, and efficiency and safety of practical application of models. In this paper, we extensively and comprehensively investigate two types of state-of-the-art learning-based approaches (sequence-based and graph-based) by conducting experiments on a recently built large-scale dataset. We investigate seven research questions from five dimensions, namely model capabilities, model interpretation, model stability, ease of use of model, and model economy. We experimentally demonstrate the priority of sequence-based models and the limited abilities of both LLM (ChatGPT) and graph-based models. We explore the types of vulnerability that learning-based models skilled in and reveal the instability of the models though the input is subtlely semantical-equivalently changed. We empirically explain what the models have learned. We summarize the pre-processing as well as requirements for easily using the models. Finally, we initially induce the vital information for economically and safely practical usage of these models.