State of What Art? A Call for Multi-Prompt LLM Evaluation
This addresses the need for more robust evaluation methods for LLM developers and users, though it is incremental as it builds on existing benchmarks.
The paper tackles the problem of brittle single-prompt evaluations for large language models (LLMs) by analyzing 6.5M instances across 20 LLMs and 39 tasks, and proposes using diverse prompts to improve robustness and provide more reliable assessments.
Recent advances in large language models (LLMs) have led to the development of various evaluation benchmarks. These benchmarks typically rely on a single instruction template for evaluating all LLMs on a specific task. In this paper, we comprehensively analyze the brittleness of results obtained via single-prompt evaluations across 6.5M instances, involving 20 different LLMs and 39 tasks from 3 benchmarks. To improve robustness of the analysis, we propose to evaluate LLMs with a set of diverse prompts instead. We discuss tailored evaluation metrics for specific use cases (e.g., LLM developers vs. developers interested in a specific downstream task), ensuring a more reliable and meaningful assessment of LLM capabilities. We then implement these criteria and conduct evaluations of multiple models, providing insights into the true strengths and limitations of current LLMs.