Inside CORE-KG: Evaluating Structured Prompting and Coreference Resolution for Knowledge Graphs
This work addresses the challenge of automated knowledge graph construction from legal documents for analyzing human smuggling networks, but it is incremental as it systematically evaluates an existing framework.
The paper tackled the problem of constructing knowledge graphs from complex legal texts about human smuggling networks, which are unstructured and contain ambiguous references, by evaluating the CORE-KG framework's components. The results showed that removing coreference resolution increased node duplication by 28.25% and noisy nodes by 4.32%, while removing structured prompts increased node duplication by 4.29% and noisy nodes by 73.33%.
Human smuggling networks are increasingly adaptive and difficult to analyze. Legal case documents offer critical insights but are often unstructured, lexically dense, and filled with ambiguous or shifting references, which pose significant challenges for automated knowledge graph (KG) construction. While recent LLM-based approaches improve over static templates, they still generate noisy, fragmented graphs with duplicate nodes due to the absence of guided extraction and coreference resolution. The recently proposed CORE-KG framework addresses these limitations by integrating a type-aware coreference module and domain-guided structured prompts, significantly reducing node duplication and legal noise. In this work, we present a systematic ablation study of CORE-KG to quantify the individual contributions of its two key components. Our results show that removing coreference resolution results in a 28.25% increase in node duplication and a 4.32% increase in noisy nodes, while removing structured prompts leads to a 4.29% increase in node duplication and a 73.33% increase in noisy nodes. These findings offer empirical insights for designing robust LLM-based pipelines for extracting structured representations from complex legal texts.