A Benchmark for the Detection of Metalinguistic Disagreements between LLMs and Knowledge Graphs
This addresses a nuanced evaluation challenge for knowledge graph engineers and NLP researchers, though it appears incremental as it builds on existing datasets and concepts.
The paper tackles the problem of distinguishing factual errors from metalinguistic disagreements in evaluating LLMs for knowledge graph tasks, proposing a benchmark for detection based on an investigation with the T-REx dataset.
Evaluating large language models (LLMs) for tasks like fact extraction in support of knowledge graph construction frequently involves computing accuracy metrics using a ground truth benchmark based on a knowledge graph (KG). These evaluations assume that errors represent factual disagreements. However, human discourse frequently features metalinguistic disagreement, where agents differ not on facts but on the meaning of the language used to express them. Given the complexity of natural language processing and generation using LLMs, we ask: do metalinguistic disagreements occur between LLMs and KGs? Based on an investigation using the T-REx knowledge alignment dataset, we hypothesize that metalinguistic disagreement does in fact occur between LLMs and KGs, with potential relevance for the practice of knowledge graph engineering. We propose a benchmark for evaluating the detection of factual and metalinguistic disagreements between LLMs and KGs. An initial proof of concept of such a benchmark is available on Github.