From Code Changes to Quality Gains: An Empirical Study in Python ML Systems with PyQu
This work addresses the lack of tools and knowledge linking code changes to quality in Python ML systems, providing a foundation for researchers, practitioners, and tool developers, though it is incremental as it builds on prior studies of code changes.
The study tackled the problem of understanding how code changes affect quality in Python-based machine learning systems by conducting a large-scale empirical analysis of 3,340 projects, introducing PyQu, a tool that identifies quality-enhancing commits with an average accuracy, precision, and recall of 0.84 and an average F1 score of 0.85, and discovering 61 impactful code changes, 41% of which were new.
In an era shaped by Generative Artificial Intelligence for code generation and the rising adoption of Python-based Machine Learning systems (MLS), software quality has emerged as a major concern. As these systems grow in complexity and importance, a key obstacle lies in understanding exactly how specific code changes affect overall quality-a shortfall aggravated by the lack of quality assessment tools and a clear mapping between ML systems code changes and their quality effects. Although prior work has explored code changes in MLS, it mostly stops at what the changes are, leaving a gap in our knowledge of the relationship between code changes and the MLS quality. To address this gap, we conducted a large-scale empirical study of 3,340 open-source Python ML projects, encompassing more than 3.7 million commits and 2.7 trillion lines of code. We introduce PyQu, a novel tool that leverages low level software metrics to identify quality-enhancing commits with an average accuracy, precision, and recall of 0.84 and 0.85 of average F1 score. Using PyQu and a thematic analysis, we identified 61 code changes, each demonstrating a direct impact on enhancing software quality, and we classified them into 13 categories based on contextual characteristics. 41% of the changes are newly discovered by our study and have not been identified by state-of-the-art Python changes detection tools. Our work offers a vital foundation for researchers, practitioners, educators, and tool developers, advancing the quest for automated quality assessment and best practices in Python-based ML software.