Bridging the KB-Text Gap: Leveraging Structured Knowledge-aware Pre-training for KBQA
This work addresses the problem of bridging the gap between text and structured knowledge for researchers and practitioners in knowledge base question answering, representing an incremental advancement with specific gains.
The paper tackles the challenge of enabling pre-trained language models to understand structured knowledge bases for question answering by proposing a Structured Knowledge-aware Pre-training method, which improves subgraph retrieval by 4.08% H@10 on the WebQSP benchmark.
Knowledge Base Question Answering (KBQA) aims to answer natural language questions with factual information such as entities and relations in KBs. However, traditional Pre-trained Language Models (PLMs) are directly pre-trained on large-scale natural language corpus, which poses challenges for them in understanding and representing complex subgraphs in structured KBs. To bridge the gap between texts and structured KBs, we propose a Structured Knowledge-aware Pre-training method (SKP). In the pre-training stage, we introduce two novel structured knowledge-aware tasks, guiding the model to effectively learn the implicit relationship and better representations of complex subgraphs. In downstream KBQA task, we further design an efficient linearization strategy and an interval attention mechanism, which assist the model to better encode complex subgraphs and shield the interference of irrelevant subgraphs during reasoning respectively. Detailed experiments and analyses on WebQSP verify the effectiveness of SKP, especially the significant improvement in subgraph retrieval (+4.08% H@10).