The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation
This addresses the challenge of inefficient software testing for developers by providing LLM-specific prompt optimization, though it is incremental as it builds on existing automated prompt optimization methods.
The paper tackles the problem of suboptimal test case generation by LLMs due to generic human-written prompts, proposing an automated method to optimize prompts tailored to each LLM, resulting in improved performance with concrete gains in test case quality.
Test cases are essential for validating the reliability and quality of software applications. Recent studies have demonstrated the capability of Large Language Models (LLMs) to generate useful test cases for given source code. However, the existing work primarily relies on human-written plain prompts, which often leads to suboptimal results since the performance of LLMs can be highly influenced by the prompts. Moreover, these approaches use the same prompt for all LLMs, overlooking the fact that different LLMs might be best suited to different prompts. Given the wide variety of possible prompt formulations, automatically discovering the optimal prompt for each LLM presents a significant challenge. Although there are methods on automated prompt optimization in the natural language processing field, they are hard to produce effective prompts for the test case generation task. First, the methods iteratively optimize prompts by simply combining and mutating existing ones without proper guidance, resulting in prompts that lack diversity and tend to repeat the same errors in the generated test cases. Second, the prompts are generally lack of domain contextual knowledge, limiting LLMs' performance in the task.