Systematic Evaluation of Knowledge Graph Repair with Large Language Models
This work addresses the need for more rigorous evaluation in knowledge graph repair, which is important for researchers and practitioners in data management and AI, though it is incremental as it builds on existing repair systems and evaluation frameworks.
The paper tackles the problem of evaluating knowledge graph repair systems by proposing a systematic method to generate constraint violations, and finds that using large language models with concise prompts that include SHACL constraints and contextual information yields the best performance.
We present a systematic approach for evaluating the quality of knowledge graph repairs with respect to constraint violations defined in shapes constraint language (SHACL). Current evaluation methods rely on \emph{ad hoc} datasets, which limits the rigorous analysis of repair systems in more general settings. Our method addresses this gap by systematically generating violations using a novel mechanism, termed violation-inducing operations (VIOs). We use the proposed evaluation framework to assess a range of repair systems which we build using large language models. We analyze the performance of these systems across different prompting strategies. Results indicate that concise prompts containing both the relevant violated SHACL constraints and key contextual information from the knowledge graph yield the best performance.