A Methodology for Investigating AI Patterns Prevalence in Software Repositories
For researchers and practitioners in AI software engineering, this work provides an initial empirical foundation for understanding the actual usage of AI patterns in code, addressing a gap in empirical data.
The paper proposes a methodology to identify AI patterns from literature and validate their prevalence in open-source repositories using active learning. The model achieves 56% accuracy and 55% recall in an 8-way classification task, significantly outperforming the 11% random baseline.
As Artificial Intelligence(AI)-based applications take off, a clear understanding of AI patterns can uplift the quality of AI applications. Many AI patterns have been proposed in the literature; however, their prevalence in real-life code has not yet been validated. Understanding the actual use of those patterns in practice can clarify our understanding both of the significance of these patterns and their utility. In this paper, we present a methodology to a) identify relevant patterns by mining the literature and then to b) validate their presence and prevalence in actual code repositories using active learning. To that end, we identify 14 AI pattern classes by mining 44 published AI pattern-related sources. Then we use an active learning approach to determine the prevalence of the most common pattern class across 100 GitHub open AI repositories. Using prevalence estimation, we propose bounds on the accuracy of the occurrences. The model achieves 56\% accuracy and 55\% recall in an 8-way classification task, significantly outperforming the 11\% random-chance baseline. Furthermore, the prevalence estimation offers usable bounds for analyzing pattern applications. This methodology provides a robust foundation to start understanding how AI patterns are used in practice, a field that currently lacks empirical data.