Towards Better Benchmark Datasets for Inductive Knowledge Graph Completion
This addresses a critical flaw in evaluation for researchers in knowledge graph completion, though it is incremental as it focuses on dataset construction rather than a new method.
The paper identifies that current inductive knowledge graph completion (KGC) benchmark datasets contain a shortcut exploitable by Personalized PageRank (PPR) scores, achieving near state-of-the-art performance, and proposes new datasets to mitigate this issue, benchmarking methods to better assess inductive KGC capabilities.
Knowledge Graph Completion (KGC) attempts to predict missing facts in a Knowledge Graph (KG). Recently, there's been an increased focus on designing KGC methods that can excel in the inductive setting, where a portion or all of the entities and relations seen in inference are unobserved during training. Numerous benchmark datasets have been proposed for inductive KGC, all of which are subsets of existing KGs used for transductive KGC. However, we find that the current procedure for constructing inductive KGC datasets inadvertently creates a shortcut that can be exploited even while disregarding the relational information. Specifically, we observe that the Personalized PageRank (PPR) score can achieve strong or near SOTA performance on most datasets. In this paper, we study the root cause of this problem. Using these insights, we propose an alternative strategy for constructing inductive KGC datasets that helps mitigate the PPR shortcut. We then benchmark multiple popular methods using the newly constructed datasets and analyze their performance. The new benchmark datasets help promote a better understanding of the capabilities and challenges of inductive KGC by removing any shortcuts that obfuscate performance. The code and dataset and can be found at https://github.com/HarryShomer/Better-Inductive-KGC.