How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset
This addresses the need for empirical data on simple bug occurrence in software engineering, though it is incremental as it builds on existing program repair templates.
The paper tackles the problem of estimating the recall of program repair techniques for simple bugs by providing the ManySStuBs4J dataset of 153,652 single-statement bug-fix changes from 1,000 Java projects, finding that about 33% of these bugs match 16 templates and occur at a frequency of about one bug per 1,600-2,500 lines of code.
Program repair is an important but difficult software engineering problem. One way to achieve acceptable performance is to focus on classes of simple bugs, such as bugs with single statement fixes, or that match a small set of bug templates. However, it is very difficult to estimate the recall of repair techniques for simple bugs, as there are no datasets about how often the associated bugs occur in code. To fill this gap, we provide a dataset of 153,652 single statement bug-fix changes mined from 1,000 popular open-source Java projects, annotated by whether they match any of a set of 16 bug templates, inspired by state-of-the-art program repair techniques. In an initial analysis, we find that about 33% of the simple bug fixes match the templates, indicating that a remarkable number of single-statement bugs can be repaired with a relatively small set of templates. Further, we find that template fitting bugs appear with a frequency of about one bug per 1,600-2,500 lines of code (as measured by the size of the project's latest version). We hope that the dataset will prove a resource for both future work in program repair and studies in empirical software engineering.