AutoWeird: Weird Translational Scoring Function Identified by Random Search
This work highlights potential flaws in evaluation protocols for knowledge graph link prediction, which could impact researchers and practitioners in the field, though it is incremental in nature.
The paper tackled the problem of evaluating scoring functions for knowledge graph link prediction by identifying AutoWeird, a scoring function found via random search that uses only tail entity and relation. It achieved top-1 performance on the ogbl-wikikg2 dataset but performed poorly on ogbl-biokg, attributed to evaluation issues and tail entity distribution.
Scoring function (SF) measures the plausibility of triplets in knowledge graphs. Different scoring functions can lead to huge differences in link prediction performances on different knowledge graphs. In this report, we describe a weird scoring function found by random search on the open graph benchmark (OGB). This scoring function, called AutoWeird, only uses tail entity and relation in a triplet to compute its plausibility score. Experimental results show that AutoWeird achieves top-1 performance on ogbl-wikikg2 data set, but has much worse performance than other methods on ogbl-biokg data set. By analyzing the tail entity distribution and evaluation protocol of these two data sets, we attribute the unexpected success of AutoWeird on ogbl-wikikg2 to inappropriate evaluation and concentrated tail entity distribution. Such results may motivate further research on how to accurately evaluate the performance of different link prediction methods for knowledge graphs.