IntelliGraphs: Datasets for Benchmarking Knowledge Graph Generation
This work addresses the need for better benchmarks in knowledge graph generation to encourage models that emphasize semantic understanding, but it is incremental as it focuses on dataset creation and baseline evaluation.
The authors tackled the problem of evaluating semantic understanding in knowledge graphs by introducing the subgraph inference task and proposing IntelliGraphs, a set of five new datasets with logical rules, and showed that baseline models based on traditional knowledge graph embeddings fail to capture the semantics.
Knowledge Graph Embedding (KGE) models are used to learn continuous representations of entities and relations. A key task in the literature is predicting missing links between entities. However, Knowledge Graphs are not just sets of links but also have semantics underlying their structure. Semantics is crucial in several downstream tasks, such as query answering or reasoning. We introduce the subgraph inference task, where a model has to generate likely and semantically valid subgraphs. We propose IntelliGraphs, a set of five new Knowledge Graph datasets. The IntelliGraphs datasets contain subgraphs with semantics expressed in logical rules for evaluating subgraph inference. We also present the dataset generator that produced the synthetic datasets. We designed four novel baseline models, which include three models based on traditional KGEs. We evaluate their expressiveness and show that these models cannot capture the semantics. We believe this benchmark will encourage the development of machine learning models that emphasize semantic understanding.