Analysis of LLMs vs Human Experts in Requirements Engineering
This addresses the problem of inefficient and costly requirements engineering in software development, offering a potential solution for improving project timelines and resource allocation, though it is incremental as it builds on existing LLM applications.
The study compared LLMs to human experts in requirements elicitation for software systems, finding that LLM-generated requirements were more aligned (+1.12) and complete (+10.2%), performed 720x faster, and cost only 0.06% of human experts.
The majority of research around Large Language Models (LLM) application to software development has been on the subject of code generation. There is little literature on LLMs' impact on requirements engineering (RE), which deals with the process of developing and verifying the system requirements. Within RE, there is a subdiscipline of requirements elicitation, which is the practice of discovering and documenting requirements for a system from users, customers, and other stakeholders. In this analysis, we compare LLM's ability to elicit requirements of a software system, as compared to that of a human expert in a time-boxed and prompt-boxed study. We found LLM-generated requirements were evaluated as more aligned (+1.12) than human-generated requirements with a trend of being more complete (+10.2%). Conversely, we found users tended to believe that solutions they perceived as more aligned had been generated by human experts. Furthermore, while LLM-generated documents scored higher and performed at 720x the speed, their cost was, on average, only 0.06% that of a human expert. Overall, these findings indicate that LLMs will play an increasingly important role in requirements engineering by improving requirements definitions, enabling more efficient resource allocation, and reducing overall project timelines.