Characterizing Knowledge Graph Tasks in LLM Benchmarks Using Cognitive Complexity Frameworks
This work addresses the need for richer interpretation and diversity in benchmark evaluation tasks for LLMs in KG applications, but it is incremental as it builds on existing frameworks without introducing new methods or data.
The authors tackled the problem of evaluating Large Language Models (LLMs) on Knowledge Graph (KG) tasks by proposing a complementary characterization approach using cognitive complexity frameworks, applied to the LLM-KG-Bench framework to highlight value distributions and identify underrepresented demands.
Large Language Models (LLMs) are increasingly used for tasks involving Knowledge Graphs (KGs), whose evaluation typically focuses on accuracy and output correctness. We propose a complementary task characterization approach using three complexity frameworks from cognitive psychology. Applying this to the LLM-KG-Bench framework, we highlight value distributions, identify underrepresented demands and motivate richer interpretation and diversity for benchmark evaluation tasks.