Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts
This addresses the challenge of understanding and improving development processes for prompt-powered software, which is used by millions, but is incremental as it builds on existing concepts of programming and mental models.
The study investigated how developers create software using natural language prompts for generative models, finding that prompt programming involves distinct practices like developing mental models of model behavior rather than code, and that even experienced developers struggle with reliable mental models.
Generative pre-trained models power intelligent software features used by millions of users controlled by developer-written natural language prompts. Despite the impact of prompt-powered software, little is known about its development process and its relationship to programming. In this work, we argue that some prompts are programs and that the development of prompts is a distinct phenomenon in programming known as "prompt programming". We develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development across a variety of contexts, models, domains, and prompt structures. We contribute 15 observations to form a preliminary understanding of current prompt programming practices. For example, rather than building mental models of code, prompt programmers develop mental models of the foundation model (FM)'s behavior on the prompt by interacting with the FM. While prior research shows that experts have well-formed mental models, we find that prompt programmers who have developed dozens of prompts still struggle to develop reliable mental models. Our observations show that prompt programming differs from traditional software development, motivating the creation of prompt programming tools and providing implications for software engineering stakeholders.