MulDE: Multi-teacher Knowledge Distillation for Low-dimensional Knowledge Graph Embeddings
This work addresses efficiency issues in knowledge graph embeddings for researchers and practitioners, offering an incremental improvement by applying distillation to reduce dimensionality while maintaining performance.
The paper tackles the problem of high training costs and storage requirements in knowledge graph embeddings by proposing MulDE, a multi-teacher knowledge distillation framework that trains low-dimensional models. The distilled 32-dimensional model achieves competitive performance compared to state-of-the-art high-dimensional methods on widely-used datasets.
Link prediction based on knowledge graph embeddings (KGE) aims to predict new triples to automatically construct knowledge graphs (KGs). However, recent KGE models achieve performance improvements by excessively increasing the embedding dimensions, which may cause enormous training costs and require more storage space. In this paper, instead of training high-dimensional models, we propose MulDE, a novel knowledge distillation framework, which includes multiple low-dimensional hyperbolic KGE models as teachers and two student components, namely Junior and Senior. Under a novel iterative distillation strategy, the Junior component, a low-dimensional KGE model, asks teachers actively based on its preliminary prediction results, and the Senior component integrates teachers' knowledge adaptively to train the Junior component based on two mechanisms: relation-specific scaling and contrast attention. The experimental results show that MulDE can effectively improve the performance and training speed of low-dimensional KGE models. The distilled 32-dimensional model is competitive compared to the state-of-the-art high-dimensional methods on several widely-used datasets.