Automated Program Repair: Emerging trends pose and expose problems for benchmarks
This highlights a critical evaluation problem for researchers and practitioners in software engineering and ML, but it is incremental as it builds on existing concerns about benchmark design.
The paper identifies that popular benchmarks for Automated Program Repair (APR) are inadequate for evaluating machine learning (ML) and large language model (LLM) methods, as they may not ensure valid or generalizable results due to issues like data contamination.
Machine learning (ML) now pervades the field of Automated Program Repair (APR). Algorithms deploy neural machine translation and large language models (LLMs) to generate software patches, among other tasks. But, there are important differences between these applications of ML and earlier work. Evaluations and comparisons must take care to ensure that results are valid and likely to generalize. A challenge is that the most popular APR evaluation benchmarks were not designed with ML techniques in mind. This is especially true for LLMs, whose large and often poorly-disclosed training datasets may include problems on which they are evaluated.