Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptions
This work addresses the efficiency of prompt engineering for researchers and practitioners, revealing that incremental improvements can be achieved by focusing on format over detailed descriptions.
The study found that the format of prompts, rather than their descriptive content, primarily drives performance improvements in large language models during in-context learning, with experiments on tasks like machine translation showing gains even with random descriptions.
With the help of in-context learning (ICL), large language models (LLMs) have achieved impressive performance across various tasks. However, the function of descriptive instructions during ICL remains under-explored. In this work, we propose an ensemble prompt framework to describe the selection criteria of multiple in-context examples, and preliminary experiments on machine translation (MT) across six translation directions confirm that this framework boosts ICL performance. But to our surprise, LLMs might not care what the descriptions actually say, and the performance gain is primarily caused by the ensemble format, since it could lead to improvement even with random descriptive nouns. We further apply this new ensemble framework on a range of commonsense, math, logical reasoning and hallucination tasks with three LLMs and achieve promising results, suggesting again that designing a proper prompt format would be much more effective and efficient than paying effort into specific descriptions.