Nalin: Learning from Runtime Behavior to Find Name-Value Inconsistencies in Jupyter Notebooks
This addresses a specific issue for developers working with Python in Jupyter notebooks, where such inconsistencies are common due to the lack of static types, and it is incremental as it builds on existing dynamic analysis methods.
The paper tackles the problem of detecting name-value inconsistencies in Python code, which can harm code understandability or cause runtime errors, by presenting Nalin, a technique that combines dynamic analysis with a neural model to classify mismatches, achieving 80% precision and 76% recall on real-world programs.
Variable names are important to understand and maintain code. If a variable name and the value stored in the variable do not match, then the program suffers from a name-value inconsistency, which is due to one of two situations that developers may want to fix: Either a correct value is referred to through a misleading name, which negatively affects code understandability and maintainability, or the correct name is bound to a wrong value, which may cause unexpected runtime behavior. Finding name-value inconsistencies is hard because it requires an understanding of the meaning of names and knowledge about the values assigned to a variable at runtime. This paper presents Nalin, a technique to automatically detect name-value inconsistencies. The approach combines a dynamic analysis that tracks assignments of values to names with a neural machine learning model that predicts whether a name and a value fit together. To the best of our knowledge, this is the first work to formulate the problem of finding coding issues as a classification problem over names and runtime values. We apply Nalin to 106,652 real-world Python programs, where meaningful names are particularly important due to the absence of statically declared types. Our results show that the classifier detects name-value inconsistencies with high accuracy, that the warnings reported by Nalin have a precision of 80% and a recall of 76% w.r.t. a ground truth created in a user study, and that our approach complements existing techniques for finding coding issues.