LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimization
This addresses the problem of costly label collection for prompt optimization in LLMs, offering a label-free method that is incremental by building on dueling-bandit and mutation techniques.
The paper tackles the challenge of automatic prompt optimization for large language models without requiring labeled data, proposing the Prompt Duel Optimizer (PDO) that uses pairwise preference feedback from an LLM judge, and it shows consistent outperformance over baselines on BIG-bench Hard and MS MARCO.
Large language models (LLMs) are highly sensitive to their input prompts, making prompt design a central challenge. While automatic prompt optimization (APO) reduces manual engineering, most approaches assume access to ground-truth references such as labeled validation data. In practice, however, collecting high-quality labels is costly and slow. We propose the Prompt Duel Optimizer (PDO), a sample-efficient framework for label-free prompt optimization. PDO formulates the problem as a dueling-bandit setting, where supervision signal comes from pairwise preference feedback provided by an LLM judge. The framework combines Double Thompson Sampling (D-TS), which prioritizes informative prompt comparisons, with Top-Performer Guided Mutation, which expands the candidate pool by mutating high-performing prompts. PDO naturally operates in label-free settings and can also incorporate partial labels to mitigate judge noise. Experiments on BIG-bench Hard (BBH) and MS MARCO show that PDO consistently outperforms baseline methods. Ablation studies further demonstrate the effectiveness of both D-TS and prompt mutation.