Automated Patch Assessment for Program Repair at Scale
This work addresses the challenge of scalable and reliable patch assessment for researchers and developers in software engineering, though it is incremental as it builds on existing RGT methods.
The paper tackles the problem of automatically assessing the correctness of patches generated by program repair systems by improving the Random testing with Ground Truth (RGT) technique, resulting in a 190% performance improvement in patch assessment accuracy. It uses a curated dataset of 638 patches from Defects4J to evaluate the method and demonstrates its reliability for overfitting analysis and enhancing external validity in program repair research.
In this paper, we do automatic correctness assessment for patches generated by program repair systems. We consider the human-written patch as ground truth oracle and randomly generate tests based on it, a technique proposed by Shamshiri et al., called Random testing with Ground Truth (RGT) in this paper. We build a curated dataset of 638 patches for Defects4J generated by 14 state-of-the-art repair systems, we evaluate automated patch assessment on this dataset. The results of this study are novel and significant: First, we improve the state of the art performance of automatic patch assessment with RGT by 190% by improving the oracle; Second, we show that RGT is reliable enough to help scientists to do overfitting analysis when they evaluate program repair systems; Third, we improve the external validity of the program repair knowledge with the largest study ever.