Comprehensive Evaluation for a Large Scale Knowledge Graph Question Answering Service
This addresses the challenge of robust evaluation for KGQA systems in industrial settings, though it is incremental as it builds on existing evaluation methods.
The paper tackles the problem of evaluating knowledge graph question answering (KGQA) systems at industry scale by introducing Chronos, a comprehensive evaluation framework that provides end-to-end and component-level metrics, scalability to diverse datasets, and performance measurement prior to release.
Question answering systems for knowledge graph (KGQA), answer factoid questions based on the data in the knowledge graph. KGQA systems are complex because the system has to understand the relations and entities in the knowledge-seeking natural language queries and map them to structured queries against the KG to answer them. In this paper, we introduce Chronos, a comprehensive evaluation framework for KGQA at industry scale. It is designed to evaluate such a multi-component system comprehensively, focusing on (1) end-to-end and component-level metrics, (2) scalable to diverse datasets and (3) a scalable approach to measure the performance of the system prior to release. In this paper, we discuss the unique challenges associated with evaluating KGQA systems at industry scale, review the design of Chronos, and how it addresses these challenges. We will demonstrate how it provides a base for data-driven decisions and discuss the challenges of using it to measure and improve a real-world KGQA system.