Joint Pre-training and Local Re-training: Transferable Representation Learning on Multi-source Knowledge Graphs
This work addresses the challenge of leveraging complementary information across different knowledge graphs for better embeddings, which is incremental as it builds on existing distillation and alignment techniques.
The paper tackles the problem of learning expressive knowledge graph embeddings from multiple sources by proposing a joint pre-training and local re-training framework, which achieves improved performance on downstream tasks through knowledge distillation and entity alignment.
In this paper, we present the ``joint pre-training and local re-training'' framework for learning and applying multi-source knowledge graph (KG) embeddings. We are motivated by the fact that different KGs contain complementary information to improve KG embeddings and downstream tasks. We pre-train a large teacher KG embedding model over linked multi-source KGs and distill knowledge to train a student model for a task-specific KG. To enable knowledge transfer across different KGs, we use entity alignment to build a linked subgraph for connecting the pre-trained KGs and the target KG. The linked subgraph is re-trained for three-level knowledge distillation from the teacher to the student, i.e., feature knowledge distillation, network knowledge distillation, and prediction knowledge distillation, to generate more expressive embeddings. The teacher model can be reused for different target KGs and tasks without having to train from scratch. We conduct extensive experiments to demonstrate the effectiveness and efficiency of our framework.