Self-attention Presents Low-dimensional Knowledge Graph Embeddings for Link Prediction
This addresses scalability issues for large knowledge bases by reducing embedding dimensions, though it is incremental as it builds on existing Transformer methods.
The paper tackles the link prediction problem in knowledge graphs by introducing a Transformer-based model that achieves expressive low-dimensional embeddings, reducing dimensionality by 66.9% compared to recent state-of-the-art competitors while maintaining comparable performance on benchmarks like WN18RR and FB15k-237.
A few models have tried to tackle the link prediction problem, also known as knowledge graph completion, by embedding knowledge graphs in comparably lower dimensions. However, the state-of-the-art results are attained at the cost of considerably increasing the dimensionality of embeddings which causes scalability issues in the case of huge knowledge bases. Transformers have been successfully used recently as powerful encoders for knowledge graphs, but available models still have scalability issues. To address this limitation, we introduce a Transformer-based model to gain expressive low-dimensional embeddings. We utilize a large number of self-attention heads as the key to applying query-dependent projections to capture mutual information between entities and relations. Empirical results on WN18RR and FB15k-237 as standard link prediction benchmarks demonstrate that our model has favorably comparable performance with the current state-of-the-art models. Notably, we yield our promising results with a significant reduction of 66.9% in the dimensionality of embeddings compared to the five best recent state-of-the-art competitors on average.