Beyond Elicitation: Provision-based Prompt Optimization for Knowledge-Intensive Tasks
This addresses the problem of inefficient prompt optimization for specialized domains, offering a novel approach with measurable gains, though it is incremental in advancing prompt optimization techniques.
The paper tackles the limitations of elicitation-based prompt optimization for knowledge-intensive tasks by proposing KPPO, a framework that integrates systematic knowledge, resulting in an average performance improvement of ~6% over baselines and up to 29% reduction in token usage.
While prompt optimization has emerged as a critical technique for enhancing language model performance, existing approaches primarily focus on elicitation-based strategies that search for optimal prompts to activate models' capabilities. These methods exhibit fundamental limitations when addressing knowledge-intensive tasks, as they operate within fixed parametric boundaries rather than providing the factual knowledge, terminology precision, and reasoning patterns required in specialized domains. To address these limitations, we propose Knowledge-Provision-based Prompt Optimization (KPPO), a framework that reformulates prompt optimization as systematic knowledge integration rather than potential elicitation. KPPO introduces three key innovations: 1) a knowledge gap filling mechanism for knowledge gap identification and targeted remediation; 2) a batch-wise candidate evaluation approach that considers both performance improvement and distributional stability; 3) an adaptive knowledge pruning strategy that balances performance and token efficiency, reducing up to 29% token usage. Extensive evaluation on 15 knowledge-intensive benchmarks from various domains demonstrates KPPO's superiority over elicitation-based methods, with an average performance improvement of ~6% over the strongest baseline while achieving comparable or lower token consumption. Code at: https://github.com/xyz9911/KPPO.