Beyond the Singular: The Essential Role of Multiple Generations in Effective Benchmark Evaluation and Analysis
This work addresses the need for more reliable benchmark evaluations in AI research, offering incremental improvements in statistical methods for assessing LLM performance.
The paper tackles the problem of unreliable benchmark evaluations for large language models (LLMs) due to overlooked randomness, proposing a hierarchical statistical model that improves score estimation accuracy and reduces variance by leveraging multiple generations, with results including a prompt-level difficulty score and a data map for error detection.
Large language models (LLMs) have demonstrated significant utilities in real-world applications, exhibiting impressive capabilities in natural language processing and understanding. Benchmark evaluations are crucial for assessing the capabilities of LLMs as they can provide a comprehensive assessment of their strengths and weaknesses. However, current evaluation methods often overlook the inherent randomness of LLMs by employing deterministic generation strategies or relying on a single random sample, resulting in unaccounted sampling variance and unreliable benchmark score estimates. In this paper, we propose a hierarchical statistical model that provides a more comprehensive representation of the benchmarking process by incorporating both benchmark characteristics and LLM randomness. We show that leveraging multiple generations improves the accuracy of estimating the benchmark score and reduces variance. We also introduce $\mathbb P\left(\text{correct}\right)$, a prompt-level difficulty score based on correct ratios, providing fine-grained insights into individual prompts. Additionally, we create a data map that visualizes difficulty and semantic prompts, enabling error detection and quality control in benchmark construction.