Submodular Evaluation Subset Selection in Automatic Prompt Optimization
This addresses a key bottleneck in prompt optimization for AI practitioners, though it is incremental as it builds on existing optimization frameworks.
The paper tackled the problem of selecting evaluation subsets for automatic prompt optimization, proposing SESS, a submodular selection method that yields better optimized prompts than random or heuristic baselines across datasets like GSM8K, MATH, and GPQA-Diamond.
Automatic prompt optimization reduces manual prompt engineering, but relies on task performance measured on a small, often randomly sampled evaluation subset as its main source of feedback signal. Despite this, how to select that evaluation subset is usually treated as an implementation detail. We study evaluation subset selection for prompt optimization from a principled perspective and propose SESS, a submodular evaluation subset selection method. We frame selection as maximizing an objective set function and show that, under mild conditions, it is monotone and submodular, enabling greedy selection with theoretical guarantees. Across GSM8K, MATH, and GPQA-Diamond, submodularly selected evaluation subsets can yield better optimized prompts than random or heuristic baselines.