Static Prediction of Runtime Errors by Learning to Execute Programs with External Resource Descriptions
This addresses the need for faster error detection in software development, offering a practical challenge for machine learning in code analysis, though it is incremental in applying interpreter-inspired models to a specific domain.
The paper tackles the problem of predicting runtime errors in programs without executing them, by introducing a dataset and an interpreter-inspired model that learns to execute programs with external resource descriptions, achieving the ability to also predict error locations despite being trained only on error presence and type.
The execution behavior of a program often depends on external resources, such as program inputs or file contents, and so cannot be run in isolation. Nevertheless, software developers benefit from fast iteration loops where automated tools identify errors as early as possible, even before programs can be compiled and run. This presents an interesting machine learning challenge: can we predict runtime errors in a "static" setting, where program execution is not possible? Here, we introduce a real-world dataset and task for predicting runtime errors, which we show is difficult for generic models like Transformers. We approach this task by developing an interpreter-inspired architecture with an inductive bias towards mimicking program executions, which models exception handling and "learns to execute" descriptions of the contents of external resources. Surprisingly, we show that the model can also predict the location of the error, despite being trained only on labels indicating the presence/absence and kind of error. In total, we present a practical and difficult-yet-approachable challenge problem related to learning program execution and we demonstrate promising new capabilities of interpreter-inspired machine learning models for code.