DETAIL Matters: Measuring the Impact of Prompt Specificity on Reasoning in Large Language Models
This work addresses the need for better prompt design strategies for researchers and practitioners using LLMs, though it is incremental as it builds on existing prompting research.
The paper tackled the problem of how prompt specificity affects reasoning in large language models, finding that more detailed prompts improve accuracy, particularly for smaller models and procedural tasks, with experiments on 30 tasks showing these gains.
Prompt design plays a critical role in the reasoning performance of large language models (LLMs), yet the impact of prompt specificity - how detailed or vague a prompt is - remains understudied. This paper introduces DETAIL, a framework for evaluating LLM performance across varying levels of prompt specificity. We generate multi-level prompts using GPT-4, quantify specificity via perplexity, and assess correctness using GPT-based semantic equivalence. Experiments on 30 novel reasoning tasks across GPT-4 and O3-mini reveal that specificity improves accuracy, especially for smaller models and procedural tasks. Our results highlight the need for adaptive prompting strategies and provide tools and data to support further research.