Through the Lens of Core Competency: Survey on Evaluation of Large Language Models
This work addresses the problem of inadequate and fragmented evaluation methods for LLMs in NLP research and industry, but it is incremental as it synthesizes existing literature rather than introducing new methods.
The paper tackles the challenge of evaluating large language models (LLMs) by surveying existing benchmarks and proposing a framework based on four core competencies: reasoning, knowledge, reliability, and safety, to organize and guide future evaluation efforts.
From pre-trained language model (PLM) to large language model (LLM), the field of natural language processing (NLP) has witnessed steep performance gains and wide practical uses. The evaluation of a research field guides its direction of improvement. However, LLMs are extremely hard to thoroughly evaluate for two reasons. First of all, traditional NLP tasks become inadequate due to the excellent performance of LLM. Secondly, existing evaluation tasks are difficult to keep up with the wide range of applications in real-world scenarios. To tackle these problems, existing works proposed various benchmarks to better evaluate LLMs. To clarify the numerous evaluation tasks in both academia and industry, we investigate multiple papers concerning LLM evaluations. We summarize 4 core competencies of LLM, including reasoning, knowledge, reliability, and safety. For every competency, we introduce its definition, corresponding benchmarks, and metrics. Under this competency architecture, similar tasks are combined to reflect corresponding ability, while new tasks can also be easily added into the system. Finally, we give our suggestions on the future direction of LLM's evaluation.