A Systematic Approach for Assessing Large Language Models' Test Case Generation Capability
This work addresses the problem of assessing LLM capabilities in software testing for developers and researchers, but it is incremental as it builds on existing interest in LLM-based test generation.
The paper tackles the lack of standardized benchmarks for evaluating large language models (LLMs) in generating test cases by proposing the GBCV approach, which systematically creates programs for assessment, and finds that GPT-4o performs better on complex structures while models struggle with arithmetic computations.
Software testing ensures the quality and reliability of software products, but manual test case creation is labor-intensive. With the rise of large language models (LLMs), there is growing interest in unit test creation with LLMs. However, effective assessment of LLM-generated test cases is limited by the lack of standardized benchmarks that comprehensively cover diverse programming scenarios. To address the assessment of LLM's test case generation ability and lacking dataset for evaluation, we propose the Generated Benchmark from Control-Flow Structure and Variable Usage Composition (GBCV) approach, which systematically generates programs used for evaluating LLMs' test generation capabilities. By leveraging basic control-flow structures and variable usage, GBCV provides a flexible framework to create a spectrum of programs ranging from simple to complex. Because GPT-4o and GPT-3-Turbo are publicly accessible models, to present real-world regular user's use case, we use GBCV to assess LLM performance on them. Our findings indicate that GPT-4o performs better on complex program structures, while all models effectively detect boundary values in simple conditions but face challenges with arithmetic computations. This study highlights the strengths and limitations of LLMs in test generation, provides a benchmark framework, and suggests directions for future improvement.