On the Shoulders of Giants: Empowering Automated Smart Contract Auditing via the GiAnt Corpus
For smart contract security researchers, this provides a large-scale, high-quality dataset to evaluate and improve automated auditing tools, addressing scalability and granularity limitations of existing datasets.
The authors propose GiANT, an automated framework that extracts structured vulnerability information from real-world audit reports to create the GiAnt Corpus, a dataset of 7,711 vulnerability findings. The dataset achieves high reliability (mean quality score 4.76/5, inter-rater agreement κ=0.88) and is used to benchmark LLMs on smart contract auditing tasks.
High-quality smart contract auditing datasets are crucial for evaluating security tools and advancing smart contract security research. Two major limitations of existing datasets are the manual-induced scalability bottleneck and the deficiency in data granularity and diversity. To address these limitations, we propose GiANT, an automated framework designed to curate smart contract auditing datasets by distilling vulnerability insights from real-world auditing reports. GiANT employs a divide-and-conquer strategy coupled with the Chain-of-Thought technique to extract structured vulnerability information from Code4rena reports, followed by an LLM-as-a-judge mechanism to perform rigorous quality assurance. To evaluate GiANT's effectiveness, we run it on 388 real-world audit reports and generate the GiAnt Corpus comprising 7,711 vulnerability findings across five severity levels. Manual assessment of the dataset demonstrates exceptional reliability in information extraction, achieving a mean quality score of $4.76\pm0.37$ (out of 5) with inter-rater agreement $κ$ of 0.88. We further validate the practicality of our dataset by benchmarking 4 state-of-the-art LLMs on vulnerability detection, code summarization, mitigation recommendation, and automated gas optimization tasks, to establish performance baselines, thereby providing a valuable data foundation for future research in automated smart contract auditing.