Confidence in Large Language Model Evaluation: A Bayesian Approach to Limited-Sample Challenges
This work addresses the challenge of reliable LLM evaluation for researchers and practitioners in scenarios with limited data, though it is incremental as it builds on existing Bayesian methods applied to a specific domain.
The authors tackled the problem of evaluating large language models (LLMs) under limited data by introducing a Bayesian approach that treats model capabilities as latent variables, resulting in superior discrimination compared to conventional methods while maintaining statistical robustness with reduced sample sizes.
Large language models (LLMs) exhibit probabilistic output characteristics, yet conventional evaluation frameworks rely on deterministic scalar metrics. This study introduces a Bayesian approach for LLM capability assessment that integrates prior knowledge through probabilistic inference, addressing limitations under limited-sample regimes. By treating model capabilities as latent variables and leveraging a curated query set to induce discriminative responses, we formalize model ranking as a Bayesian hypothesis testing problem over mutually exclusive capability intervals. Experimental evaluations with GPT-series models demonstrate that the proposed method achieves superior discrimination compared to conventional evaluation methods. Results indicate that even with reduced sample sizes, the approach maintains statistical robustness while providing actionable insights, such as probabilistic statements about a model's likelihood of surpassing specific baselines. This work advances LLM evaluation methodologies by bridging Bayesian inference with practical constraints in real-world deployment scenarios.