Are Missing Links Predictable? An Inferential Benchmark for Knowledge Graph Completion
This work addresses the need for better benchmarks in knowledge graph completion for researchers, though it is incremental as it builds on existing datasets with methodological improvements.
The authors introduced InferWiki, a Knowledge Graph Completion dataset designed to improve inferential ability by ensuring test samples are predictable from training data using rule-guided splits, and by incorporating open-world assumptions and diverse inference patterns. Experiments on two settings of InferWiki and CoDEx showed performance gaps across assumptions and patterns, highlighting dataset quality and research challenges.
We present InferWiki, a Knowledge Graph Completion (KGC) dataset that improves upon existing benchmarks in inferential ability, assumptions, and patterns. First, each testing sample is predictable with supportive data in the training set. To ensure it, we propose to utilize rule-guided train/test generation, instead of conventional random split. Second, InferWiki initiates the evaluation following the open-world assumption and improves the inferential difficulty of the closed-world assumption, by providing manually annotated negative and unknown triples. Third, we include various inference patterns (e.g., reasoning path length and types) for comprehensive evaluation. In experiments, we curate two settings of InferWiki varying in sizes and structures, and apply the construction process on CoDEx as comparative datasets. The results and empirical analyses demonstrate the necessity and high-quality of InferWiki. Nevertheless, the performance gap among various inferential assumptions and patterns presents the difficulty and inspires future research direction. Our datasets can be found in https://github.com/TaoMiner/inferwiki