Can the capability of Large Language Models be described by human ability? A Meta Study
This work addresses the debate over LLMs' human-like capabilities for researchers and users, providing empirical insights but is incremental in nature.
The study tackled the problem of characterizing Large Language Models' capabilities in relation to human abilities by analyzing performance data from over 80 models across 37 benchmarks, finding that certain capabilities of smaller models can be described using human metrics, abilities are less correlated in LLMs than in humans, and capabilities vary with model scale.
Users of Large Language Models (LLMs) often perceive these models as intelligent entities with human-like capabilities. However, the extent to which LLMs' capabilities truly approximate human abilities remains a topic of debate. In this paper, to characterize the capabilities of LLMs in relation to human capabilities, we collected performance data from over 80 models across 37 evaluation benchmarks. The evaluation benchmarks are categorized into 6 primary abilities and 11 sub-abilities in human aspect. Then, we then clustered the performance rankings into several categories and compared these clustering results with classifications based on human ability aspects. Our findings lead to the following conclusions: 1. We have confirmed that certain capabilities of LLMs with fewer than 10 billion parameters can indeed be described using human ability metrics; 2. While some abilities are considered interrelated in humans, they appear nearly uncorrelated in LLMs; 3. The capabilities possessed by LLMs vary significantly with the parameter scale of the model.