HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning
This work addresses a domain-specific problem for software testing and code generation, offering an incremental improvement over existing methods.
The paper tackles the limited diversity and debugging information in existing LLM-based test generation methods by proposing HarnessLLM, a two-stage training pipeline that enables LLMs to generate harness code for testing, resulting in improved bug finding and testing strategy diversity.
Existing LLM-based automatic test generation methods mainly produce input and expected output pairs to categorize the intended behavior of correct programs. Although straightforward, these methods have limited diversity in generated tests and cannot provide enough debugging information. We propose HarnessLLM, a two-stage training pipeline that enables LLMs to write harness code for testing. Particularly, LLMs generate code that synthesizes inputs and validates the observed outputs, allowing complex test cases and flexible output validation such as invariant checking. To achieve this, we train LLMs with SFT followed by RLVR with a customized reward design. Experiments show that HarnessLLM outperforms input-output-based testing in bug finding and testing strategy diversity. HarnessLLM further benefits the code generation performance through test-time scaling with our generated test cases as inference-phase validation. Our code is available at https://github.com/UCSB-NLP-Chang/HarnessLLM.git.