Reflection in the Dark: Exposing and Escaping the Black Box in Reflective Prompt Optimization
This addresses the issue of systematic failure and lack of interpretability in APO for LLM users, offering a novel solution with significant performance gains.
The paper tackled the problem of black-box and uninterpretable trajectories in reflective automatic prompt optimization (APO) methods, which can degrade performance, such as reducing accuracy from 23.81% to 13.50% on GSM8K with a defective seed. It proposed VISTA, a multi-agent APO framework that recovered accuracy to 87.57% on the same seed and outperformed baselines across benchmarks.
Automatic prompt optimization (APO) has emerged as a powerful paradigm for improving LLM performance without manual prompt engineering. Reflective APO methods such as GEPA iteratively refine prompts by diagnosing failure cases, but the optimization process remains black-box and label-free, leading to uninterpretable trajectories and systematic failure. We identify and empirically demonstrate four limitations: on GSM8K with a defective seed, GEPA degrades accuracy from 23.81% to 13.50%. We propose VISTA, a multi-agent APO framework that decouples hypothesis generation from prompt rewriting, enabling semantically labeled hypotheses, parallel minibatch verification, and interpretable optimization trace. A two-layer explore-exploit mechanism combining random restart and epsilon-greedy sampling further escapes local optima. VISTA recovers accuracy to 87.57% on the same defective seed and consistently outperforms baselines across all conditions on GSM8K and AIME2025.