Prompt Optimization Is a Coin Flip: Diagnosing When It Helps in Compound AI Systems
For practitioners building compound AI systems, this work diagnoses when prompt optimization is beneficial, turning a random outcome into an informed decision.
Prompt optimization in compound AI systems often fails, with 49% of runs on Claude Haiku and even higher failure rates on Amazon Nova Lite performing worse than zero-shot. The paper shows optimization helps only when tasks have exploitable output structure, and provides a diagnostic to predict when optimization is worthwhile.
Prompt optimization in compound AI systems is statistically indistinguishable from a coin flip: across 72 optimization runs on Claude Haiku (6 methods $\times$ 4 tasks $\times$ 3 repeats), 49% score below zero-shot; on Amazon Nova Lite, the failure rate is even higher. Yet on one task, all six methods improve over zero-shot by up to $+6.8$ points. What distinguishes success from failure? We investigate with 18,000 grid evaluations and 144 optimization runs, testing two assumptions behind end-to-end optimization tools like TextGrad and DSPy: (A) individual prompts are worth optimizing, and (B) agent prompts interact, requiring joint optimization. Interaction effects are never significant ($p > 0.52$, all $F < 1.0$), and optimization helps only when the task has exploitable output structure -- a format the model can produce but does not default to. We provide a two-stage diagnostic: an \$80 ANOVA pre-test for agent coupling, and a 10-minute headroom test that predicts whether optimization is worthwhile -- turning a coin flip into an informed decision.