Enhancing Metacognitive AI: Knowledge-Graph Population with Graph-Theoretic LLM Enrichment
This work addresses the lack of metacognitive self-correction in LLM applications, offering a proof-of-concept for automated knowledge repair.
MetaKGEnrich is a pipeline that enables LLMs to autonomously detect and fill knowledge gaps by building knowledge graphs, identifying sparse regions via graph metrics, and retrieving targeted web evidence. It improved answer quality in 80-87% of queries across three QA datasets.
Metacognition-the ability to monitor one's own knowledge state, spot gaps, and autonomously fill them--remains largely absent from modern AI. Here, we present MetaKGEnrich, a fully automated pipeline that endows large language model (LLM) applications with self-directed knowledge repair. The system (i) builds knowledge graphs from a seed query, (ii) detects sparse regions via seven graph metrics, (iii) has GPT-4o generate targeted questions, (iv) retrieves web evidence with Tavily and ingests it into Neo4j, and (v) re-answers the query with GraphRAG for GPT-4 to evaluate improvement. Tested on 30 queries from each of three widely-used datasets: Google Research Natural Questions, MS MARCO, and Hot-potQA. MetaKGEnrich improved answer quality in 80% of HotpotQA questions, 87% of Google Research Natural Questions and 83% of MS MARCO questions, while preserving well-supported regions. This proof of concept demonstrates how topological self-diagnosis plus targeted retrieval can advance AI toward humanlike metacognitive learning.