On Evaluating Embedding Models for Knowledge Base Completion
This work highlights a critical issue in evaluating knowledge base completion methods, which could impact researchers and practitioners in web search and mining by revealing that current models may not be as effective as previously thought.
The paper argues that current evaluation protocols for embedding models in knowledge base completion are flawed, as they are more suited for question answering, and shows that under a different prediction task, these models perform poorly even on easy datasets, especially compared to a simple rule-based baseline.
Knowledge bases contribute to many web search and mining tasks, yet they are often incomplete. To add missing facts to a given knowledge base, various embedding models have been proposed in the recent literature. Perhaps surprisingly, relatively simple models with limited expressiveness often performed remarkably well under today's most commonly used evaluation protocols. In this paper, we explore whether recent models work well for knowledge base completion and argue that the current evaluation protocols are more suited for question answering rather than knowledge base completion. We show that when focusing on a different prediction task for evaluating knowledge base completion, the performance of current embedding models is unsatisfactory even on datasets previously thought to be too easy. This is especially true when embedding models are compared against a simple rule-based baseline. This work indicates the need for more research into the embedding models and evaluation protocols for knowledge base completion.