KGEval: Estimating Accuracy of Automatically Constructed Knowledge Graphs
This addresses a critical gap for researchers and practitioners working with large-scale knowledge graphs, though it is incremental as it builds on existing methods for accuracy estimation.
The paper tackles the problem of estimating the accuracy of automatically constructed knowledge graphs, which is challenging due to their size and diversity, and proposes KGEval, a method that uses coupling constraints and crowdsourcing to infer correctness with fewer human evaluations, achieving more accurate estimates compared to baselines.
Automatic construction of large knowledge graphs (KG) by mining web-scale text datasets has received considerable attention recently. Estimating accuracy of such automatically constructed KGs is a challenging problem due to their size and diversity. This important problem has largely been ignored in prior research we fill this gap and propose KGEval. KGEval binds facts of a KG using coupling constraints and crowdsources the facts that infer correctness of large parts of the KG. We demonstrate that the objective optimized by KGEval is submodular and NP-hard, allowing guarantees for our approximation algorithm. Through extensive experiments on real-world datasets, we demonstrate that KGEval is able to estimate KG accuracy more accurately compared to other competitive baselines, while requiring significantly lesser number of human evaluations.