Rethinking Technology Stack Selection with AI Coding Proficiency
For software development teams using LLM assistants, this work highlights a new factor (AI coding proficiency) that should be considered in technology selection to avoid increased debugging effort and technical debt.
The paper introduces the concept of AI coding proficiency, measuring how well LLMs can use a technology to generate quality code. An empirical study across 170 libraries and six LLMs found up to 84% differences in code quality scores, impacting engineering costs and threatening technological diversity.
Large language models (LLMs) are now an integral part of software development workflows and are reshaping the whole process. However, existing technology selection methods mainly focus on the inherent attributes of technologies, overlooking whether the LLM can effectively leverage the chosen technology. Therefore, teams using LLM assistants risk choosing technologies that cannot be used effectively by LLMs, yielding high debugging effort and mounting technical debt. We foresee a practical question in the LLM era, is a technology ready for AI-assisted development? In this paper, we first propose the concept, AI coding proficiency, the degree to which LLMs can utilize a given technology to generate high-quality code snippets. We then conduct the first large-scale empirical study examining AI coding proficiency across 170 third-party libraries and six LLMs. Our findings reveal that libraries with similar functionalities can exhibit up to 84% differences in the quality score of LLM-generated code. These gaps can be translated into real engineering costs and steer developer choices toward a narrow set of technologies, threatening technological diversity in the ecosystem. We call on the community to integrate AI coding proficiency into technology selection frameworks and develop mitigation strategies, preserving competitive balance in AI-driven development.