How Do Software Professionals Evaluate AI-Generated Code? (Registered Report)
For software professionals and tool developers, this study addresses the gap in understanding evaluation practices for AI-generated code, though it is a qualitative theory-building effort without quantitative results.
This registered report investigates how software professionals evaluate AI-generated code, using a constructivist grounded theory approach with surveys and interviews. The study aims to build a theory grounded in practitioners' accounts, with initial survey data collected and plans to interview 20-50 professionals.
Recent advances in generative AI tools have significantly changed how software professionals write, evaluate, and interact with code. Generative AI tools such as GitHub Copilot, ChatGPT, and Claude are increasingly being integrated into everyday workflows. Despite the growing adoption of and reliance on these tools, it remains unclear as to how software professionals evaluate the code they generate. To explore this topic, we will conduct a constructivist grounded theory study that incorporates a survey, semi-structured interviews, and laddering interviews. With the initial survey data collection complete, we aim to interview 20--50 software professionals iteratively until theoretical saturation is achieved. This research aims to build a theory of how software professionals evaluate AI-generated code, grounded in their accounts of evaluative practices, perceptions, and preferences.