Evaluating the Knowledge Base Completion Potential of GPT
This work addresses the challenge of incomplete knowledge bases for search engines and applications, but it is incremental as it builds on prior methods with improved results.
The paper tackles the problem of using language models for unsupervised knowledge base completion on Wikidata, finding that GPT models do not achieve fully convincing results but provide solid improvements, with GPT-3 enabling an extension of Wikidata by 27 million facts at 90% precision.
Structured knowledge bases (KBs) are an asset for search engines and other applications, but are inevitably incomplete. Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC), yet, their ability to do this at scale and with high accuracy remains an open question. Prior experimental studies mostly fall short because they only evaluate on popular subjects, or sample already existing facts from KBs. In this work, we perform a careful evaluation of GPT's potential to complete the largest public KB: Wikidata. We find that, despite their size and capabilities, models like GPT-3, ChatGPT and GPT-4 do not achieve fully convincing results on this task. Nonetheless, they provide solid improvements over earlier approaches with smaller LMs. In particular, we show that, with proper thresholding, GPT-3 enables to extend Wikidata by 27M facts at 90% precision.