Rethinking Issue Resolution for AI/ML Systems
For researchers and practitioners maintaining AI/ML systems, this paper identifies gaps in existing issue resolution frameworks and provides a foundation for developing specialized tools and practices.
The authors conducted a qualitative study of 100 issue reports and pull requests from TensorFlow, scikit-learn, MLflow, and AutoGPT to identify unique characteristics of issue resolution in AI/ML systems, such as stochastic behavior, experimentation-driven workflows, and heterogeneous artifacts. They propose a vision for tailored issue resolution frameworks and discuss needed research directions and tooling.
We advocate for AI/ML issue resolution frameworks tailored to maintenance workflows and the nature of modern AI/ML systems. Existing issue resolution frameworks largely emerged for traditional software maintenance practices and do not explicitly account for characteristics common in AI/ML systems, such as stochastic behavior, experimentation-driven workflows, and heterogeneous artifacts beyond source code. To identify the unique characteristics of issue resolution in AI/ML systems and motivate the need for tailored frameworks, we conducted a qualitative study of issue resolution workflows documented in 100 issue reports and pull requests across four widely used AI/ML systems: TensorFlow, scikit-learn, MLflow, and AutoGPT. Our findings suggest that issue resolution in AI/ML systems involves: recurring AI/ML-related activities that span multiple resolution stages; iterative experimentation and adaptive verification; and coordinated changes across artifacts such as datasets, prompts, and model configurations. We also observed challenges related to reproducibility, nondeterministic behavior, and artifact coordination. Building on these findings, we present a vision for AI/ML issue resolution frameworks and discuss research directions and tooling support needed to realize this vision.