Beyond Benchmarking: A New Paradigm for Evaluation and Assessment of Large Language Models
This addresses the problem of inadequate evaluation and improvement guidance for LLM developers and researchers, though it appears incremental as it builds on existing benchmarking concepts.
The paper tackles the limitations of current benchmarks for large language models (LLMs), such as restricted content and lack of optimization guidance, by proposing a new paradigm called Benchmarking-Evaluation-Assessment that shifts evaluation from an 'examination room' to a 'hospital' model, using task-solving for deep problem attribution and optimization recommendations.
In current benchmarks for evaluating large language models (LLMs), there are issues such as evaluation content restriction, untimely updates, and lack of optimization guidance. In this paper, we propose a new paradigm for the measurement of LLMs: Benchmarking-Evaluation-Assessment. Our paradigm shifts the "location" of LLM evaluation from the "examination room" to the "hospital". Through conducting a "physical examination" on LLMs, it utilizes specific task-solving as the evaluation content, performs deep attribution of existing problems within LLMs, and provides recommendation for optimization.