KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models
This addresses the need for more interpretable evaluation in knowledge graph completion research, though it is incremental as it builds on existing evaluation methods.
The authors tackled the problem that averaged single-score metrics for knowledge graph completion models fail to reveal what models have learned or failed to learn, and they proposed KGxBoard, an interactive framework for fine-grained evaluation on meaningful data subsets, enabling discoveries impossible with standard metrics.
Knowledge Graphs (KGs) store information in the form of (head, predicate, tail)-triples. To augment KGs with new knowledge, researchers proposed models for KG Completion (KGC) tasks such as link prediction; i.e., answering (h; p; ?) or (?; p; t) queries. Such models are usually evaluated with averaged metrics on a held-out test set. While useful for tracking progress, averaged single-score metrics cannot reveal what exactly a model has learned -- or failed to learn. To address this issue, we propose KGxBoard: an interactive framework for performing fine-grained evaluation on meaningful subsets of the data, each of which tests individual and interpretable capabilities of a KGC model. In our experiments, we highlight the findings that we discovered with the use of KGxBoard, which would have been impossible to detect with standard averaged single-score metrics.