ConDefects: A New Dataset to Address the Data Leakage Concern for LLM-based Fault Localization and Program Repair
This addresses the issue of misleading performance metrics for researchers and developers in software engineering, particularly for fault localization and program repair methods, though it is incremental as it focuses on dataset creation rather than novel algorithmic advances.
The authors tackled the problem of data leakage in LLM-based fault localization and program repair by introducing ConDefects, a new dataset of 1,254 Java and 1,625 Python faulty programs sourced from AtCoder between 2021 and 2023, which eliminates overlap with LLM training data to provide more reliable benchmarks.
With the growing interest on Large Language Models (LLMs) for fault localization and program repair, ensuring the integrity and generalizability of the LLM-based methods becomes paramount. The code in existing widely-adopted benchmarks for these tasks was written before the the bloom of LLMs and may be included in the training data of existing popular LLMs, thereby suffering from the threat of data leakage, leading to misleadingly optimistic performance metrics. To address this issue, we introduce "ConDefects", a novel dataset of real faults meticulously curated to eliminate such overlap. ConDefects contains 1,254 Java faulty programs and 1,625 Python faulty programs. All these programs are sourced from the online competition platform AtCoder and were produced between October 2021 and September 2023. We pair each fault with fault locations and the corresponding repaired code versions, making it tailored for in fault localization and program repair related research. We also provide interfaces for selecting subsets based on different time windows and coding task difficulties. While inspired by LLM-based tasks, ConDefects can be adopted for benchmarking ALL types of fault localization and program repair methods. The dataset is publicly available, and a demo video can be found at https://www.youtube.com/watch?v=22j15Hj5ONk.