SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questions
This work addresses the challenge of handling complex, real-world questions in KBQA for users of large knowledge bases like Wikidata, though it is incremental as it builds on existing LLM and KBQA methods.
The authors tackled the problem of Knowledge Base Question Answering (KBQA) by introducing the SPINACH dataset, which captures complex real-world questions from Wikidata, and an in-context learning agent that achieves state-of-the-art results, such as 31.0% improvement on QALD-7 and outperforming GPT-4-based agents by at least 38.1% on their dataset.
Large Language Models (LLMs) have led to significant improvements in the Knowledge Base Question Answering (KBQA) task. However, datasets used in KBQA studies do not capture the true complexity of KBQA tasks. They either have simple questions, use synthetically generated logical forms, or are based on small knowledge base (KB) schemas. We introduce the SPINACH dataset, an expert-annotated KBQA dataset collected from discussions on Wikidata's "Request a Query" forum with 320 decontextualized question-SPARQL pairs. The complexity of these in-the-wild queries calls for a KBQA system that can dynamically explore large and often incomplete schemas and reason about them, as it is infeasible to create a comprehensive training dataset. We also introduce an in-context learning KBQA agent, also called SPINACH, that mimics how a human expert would write SPARQLs to handle challenging questions. SPINACH achieves a new state of the art on the QALD-7, QALD-9 Plus and QALD-10 datasets by 31.0%, 27.0%, and 10.0% in $F_1$, respectively, and coming within 1.6% of the fine-tuned LLaMA SOTA model on WikiWebQuestions. On our new SPINACH dataset, the SPINACH agent outperforms all baselines, including the best GPT-4-based KBQA agent, by at least 38.1% in $F_1$.