LLM-Based Invariant Testing for Software Functional Bugs
For software developers, LISA automates the detection of functional bugs that do not cause crashes, reducing manual testing effort.
LISA uses LLMs to generate API sequences and program invariants for detecting functional bugs in software libraries, achieving higher bug-detection rates and competitive code coverage compared to fuzzing and prior LLM-based methods.
Manually writing unit tests to uncover functional bugs in software libraries is not only time-consuming but also requires a deep understanding of the intended semantics of the APIs. Heuristic-based test generation methods suffer from low usability because they cannot reason about program semantics or interpret source code and documentation as humans do. Traditional fuzzing techniques like OSS-Fuzz often rely on crashes to detect bugs, but functional bugs do not always cause crashes. To overcome these limitations, we present LISA, a novel LLM-based invariant testing framework for software functional bugs. LISA iteratively generates API sequences and program invariants guided by API n-gram feedback, achieving higher bug-detection rates and competitive code coverage compared with both fuzzing and prior LLM-based test generation approaches, and reporting each finding as a high-confidence bug candidate for developer confirmation.