APITestGenie: Generating Web API Tests from Requirements and API Specifications with LLMs
This addresses the problem of reducing manual effort and improving test alignment for software developers and testers in industries relying on Web APIs, though it is incremental as it builds on existing LLM and RAG techniques.
The paper tackled the manual and error-prone task of creating Web API tests by introducing APITestGenie, a tool that automatically generates API integration tests from requirements and specifications using LLMs, achieving 89% valid test generation for real-world APIs and uncovering unknown defects.
Modern software systems rely heavily on Web APIs, yet creating meaningful and executable test scripts remains a largely manual, time-consuming, and error-prone task. In this paper, we present APITestGenie, a novel tool that leverages Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), and prompt engineering to automatically generate API integration tests directly from business requirements and OpenAPI specifications. We evaluated APITestGenie on 10 real-world APIs, including 8 APIs comprising circa 1,000 live endpoints from an industrial partner in the automotive domain. The tool was able to generate syntactically and semantically valid test scripts for 89\% of the business requirements under test after at most three attempts. Notably, some generated tests revealed previously unknown defects in the APIs, including integration issues between endpoints. Statistical analysis identified API complexity and level of detail in business requirements as primary factors influencing success rates, with the level of detail in API documentation also affecting outcomes. Feedback from industry practitioners confirmed strong interest in adoption, substantially reducing the manual effort in writing acceptance tests, and improving the alignment between tests and business requirements.