Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index
This addresses the need for cost-effective prompt engineering in resource-constrained scenarios for end-users and researchers, though it is incremental as it builds on existing prompting techniques.
The paper tackles the problem of evaluating prompt engineering techniques beyond accuracy by introducing the Economical Prompting Index (EPI), a metric that balances accuracy with token consumption based on user cost concerns, and finds that simpler methods like Chain-of-Thought (EPI 0.72) outperform complex ones like Self-Consistency (EPI 0.64) in cost-effectiveness.
As prompt engineering research rapidly evolves, evaluations beyond accuracy are crucial for developing cost-effective techniques. We present the Economical Prompting Index (EPI), a novel metric that combines accuracy scores with token consumption, adjusted by a user-specified cost concern level to reflect different resource constraints. Our study examines 6 advanced prompting techniques, including Chain-of-Thought, Self-Consistency, and Tree of Thoughts, across 10 widely-used language models and 4 diverse datasets. We demonstrate that approaches such as Self-Consistency often provide statistically insignificant gains while becoming cost-prohibitive. For example, on high-performing models like Claude 3.5 Sonnet, the EPI of simpler techniques like Chain-of-Thought (0.72) surpasses more complex methods like Self-Consistency (0.64) at slight cost concern levels. Our findings suggest a reevaluation of complex prompting strategies in resource-constrained scenarios, potentially reshaping future research priorities and improving cost-effectiveness for end-users.