Leveraging LLMs in Scholarly Knowledge Graph Question Answering
This work addresses the challenge of improving question answering accuracy in scholarly domains, though it is incremental as it builds on existing methods like BERT and LLMs.
The paper tackles the problem of answering bibliographic natural language questions by developing a scholarly Knowledge Graph Question Answering system that leverages a large language model in a few-shot manner, achieving an F1 score of 99.0% on the SciQA benchmark.
This paper presents a scholarly Knowledge Graph Question Answering (KGQA) that answers bibliographic natural language questions by leveraging a large language model (LLM) in a few-shot manner. The model initially identifies the top-n similar training questions related to a given test question via a BERT-based sentence encoder and retrieves their corresponding SPARQL. Using the top-n similar question-SPARQL pairs as an example and the test question creates a prompt. Then pass the prompt to the LLM and generate a SPARQL. Finally, runs the SPARQL against the underlying KG - ORKG (Open Research KG) endpoint and returns an answer. Our system achieves an F1 score of 99.0%, on SciQA - one of the Scholarly-QALD-23 challenge benchmarks.