DualDE: Dually Distilling Knowledge Graph Embedding for Faster and Cheaper Reasoning
This addresses storage and computing challenges for resource-limited or time-constrained applications in KG reasoning, representing an incremental improvement through distillation techniques.
The paper tackles the problem of high-dimensional Knowledge Graph Embeddings (KGEs) being resource-intensive by proposing DualDE, a knowledge distillation method that reduces embedding parameters by 7-15 times and increases inference speed by 2-6 times while maintaining high performance.
Knowledge Graph Embedding (KGE) is a popular method for KG reasoning and training KGEs with higher dimension are usually preferred since they have better reasoning capability. However, high-dimensional KGEs pose huge challenges to storage and computing resources and are not suitable for resource-limited or time-constrained applications, for which faster and cheaper reasoning is necessary. To address this problem, we propose DualDE, a knowledge distillation method to build low-dimensional student KGE from pre-trained high-dimensional teacher KGE. DualDE considers the dual-influence between the teacher and the student. In DualDE, we propose a soft label evaluation mechanism to adaptively assign different soft label and hard label weights to different triples, and a two-stage distillation approach to improve the student's acceptance of the teacher. Our DualDE is general enough to be applied to various KGEs. Experimental results show that our method can successfully reduce the embedding parameters of a high-dimensional KGE by 7 times - 15 times and increase the inference speed by 2 times - 6 times while retaining a high performance. We also experimentally prove the effectiveness of our soft label evaluation mechanism and two-stage distillation approach via ablation study.