Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs
This addresses the problem of high evaluation costs for LLM developers and researchers by providing a systematic method to reduce test data needs, though it is incremental as it builds on existing evaluation practices.
The paper tackles the challenge of evaluating large language models (LLMs) by introducing a certifiable and cost-efficient framework that adapts to different evaluation objectives and provides confidence intervals for true performance values, with experiments showing it saves 20% to 40% of test points while maintaining comparable error levels and offering a 95% confidence guarantee.
As foundation models continue to scale, the size of trained models grows exponentially, presenting significant challenges for their evaluation. Current evaluation practices involve curating increasingly large datasets to assess the performance of large language models (LLMs). However, there is a lack of systematic analysis and guidance on determining the sufficiency of test data or selecting informative samples for evaluation. This paper introduces a certifiable and cost-efficient evaluation framework for LLMs. Our framework adapts to different evaluation objectives and outputs confidence intervals that contain true values with high probability. We use ``test sample complexity'' to quantify the number of test points needed for a certifiable evaluation and derive tight bounds on test sample complexity. Based on the developed theory, we develop a partition-based algorithm, named Cer-Eval, that adaptively selects test points to minimize the cost of LLM evaluation. Real-world experiments demonstrate that Cer-Eval can save 20% to 40% test points across various benchmarks, while maintaining an estimation error level comparable to the current evaluation process and providing a 95% confidence guarantee.