On the Viability of Requirements Generation From Code: An Experience Report
For researchers in Requirements Engineering, this experience report highlights the current limitations of LLMs for generating realistic code-to-requirements datasets, indicating that the approach is not yet viable.
The authors attempted to use LLMs to generate requirements from source code for empirical RE research, but found that LLMs could not reliably generate non-implemented requirements, high-quality requirements, or synthetic requirements smells, and that human-in-the-loop detection of smells is insufficient. They conclude that LLM-based requirements generation is not yet viable without extensive human supervision.
Empirical research in Requirements Engineering is hampered by a lack of adequate datasets that pair source code with corresponding requirements. A tempting route to addressing this lack is the use of Large Language Models to synthesize requirements from existing code bases. We investigate this question by evaluating an LLM-based and RAG-supported agentic approach that generates requirements from source code, verifies their implementation status relying on a human-in-the-loop, and synthetically introduces requirements smells and non-implemented requirements. Our goal was to create datasets that mimic reality and foster empirical RE research. However, during the study, various problems arose, leading to this experience report. Contrary to our initial hypotheses, LLMs were unable to (i) generate non-implemented requirements reliably, (ii) generate high quality requirements, and (iii) reliably introduce synthetic requirements smells. Furthermore, neither an LLM nor a single human-in-the-loop suffices to detect requirements smells reliably. These findings suggest that the generation of code-to-requirements datasets using LLMs is not yet viable and requires human supervision, especially for quality assurance. We critically reflect on our lessons learned and draw relevant conclusions for both researchers and practitioners.