A Framework for Robust Cognitive Evaluation of LLMs
This provides a tool for researchers in cognitive science and AI to systematically assess LLM cognition, though it is incremental as it builds on existing evaluation approaches.
The authors tackled the lack of standardized methods for evaluating cognitive abilities in large language models by developing CognitivEval, a framework that uses automatic prompt permutations and collects both generations and probability estimates to improve robustness, and they applied it to replicate five classic cognitive science experiments to profile state-of-the-art LLMs.
Emergent cognitive abilities in large language models (LLMs) have been widely observed, but their nature and underlying mechanisms remain poorly understood. A growing body of research draws on cognitive science to investigate LLM cognition, but standard methodologies and experimen-tal pipelines have not yet been established. To address this gap we develop CognitivEval, a framework for systematically evaluating the artificial cognitive capabilities of LLMs, with a particular emphasis on robustness in response collection. The key features of CognitivEval include: (i) automatic prompt permutations, and (ii) testing that gathers both generations and model probability estimates. Our experiments demonstrate that these features lead to more robust experimental outcomes. Using CognitivEval, we replicate five classic experiments in cognitive science, illustrating the framework's generalizability across various experimental tasks and obtaining a cognitive profile of several state of the art LLMs. CognitivEval will be released publicly to foster broader collaboration within the cognitive science community.