A Unified Evaluation-Instructed Framework for Query-Dependent Prompt Optimization
This work addresses the challenge of unreliable and uninterpretable prompt optimization for users in dynamic AI applications, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackles the problem of prompt optimization in complex user scenarios by establishing a systematic prompt evaluation framework and developing an execution-free evaluator that predicts multi-dimensional quality scores, which then instructs a metric-aware optimizer to rewrite prompts. The approach consistently surpasses baselines across eight datasets and three backbone models, achieving the strongest accuracy in predicting prompt performance.
Most prompt-optimization methods refine a single static template, making them ineffective in complex and dynamic user scenarios. Existing query-dependent approaches rely on unstable textual feedback or black-box reward models, providing weak and uninterpretable optimization signals. More fundamentally, prompt quality itself lacks a unified, systematic definition, resulting in fragmented and unreliable evaluation signals. Our approach first establishes a performance-oriented, systematic, and comprehensive prompt evaluation framework. Furthermore, we develop and finetune an execution-free evaluator that predicts multi-dimensional quality scores directly from text. The evaluator then instructs a metric-aware optimizer that diagnoses failure modes and rewrites prompts in an interpretable, query-dependent manner. Our evaluator achieves the strongest accuracy in predicting prompt performance, and the evaluation-instructed optimization consistently surpass both static-template and query-dependent baselines across eight datasets and on three backbone models. Overall, we propose a unified, metric-grounded perspective on prompt quality, and demonstrated that our evaluation-instructed optimization pipeline delivers stable, interpretable, and model-agnostic improvements across diverse tasks.