SPBERT: An Efficient Pre-training BERT on SPARQL Queries for Question Answering over Knowledge Graphs
This addresses the challenge of improving question answering for users of knowledge graphs, though it appears incremental as it builds on existing BERT and transformer methods.
The paper tackles the problem of question answering over knowledge graphs by pre-training a BERT model on SPARQL queries, resulting in state-of-the-art BLEU scores on tasks like SPARQL query construction and answer verbalization.
In this paper, we propose SPBERT, a transformer-based language model pre-trained on massive SPARQL query logs. By incorporating masked language modeling objectives and the word structural objective, SPBERT can learn general-purpose representations in both natural language and SPARQL query language. We investigate how SPBERT and encoder-decoder architecture can be adapted for Knowledge-based QA corpora. We conduct exhaustive experiments on two additional tasks, including SPARQL Query Construction and Answer Verbalization Generation. The experimental results show that SPBERT can obtain promising results, achieving state-of-the-art BLEU scores on several of these tasks.