HittER: Hierarchical Transformers for Knowledge Graph Embeddings
This addresses knowledge graph embedding for link prediction and question answering, with incremental improvements in performance.
The paper tackles the problem of learning entity and relation representations in knowledge graphs by proposing HittER, a hierarchical Transformer model that achieves new state-of-the-art results on multiple link prediction datasets.
This paper examines the challenging problem of learning representations of entities and relations in a complex multi-relational knowledge graph. We propose HittER, a Hierarchical Transformer model to jointly learn Entity-relation composition and Relational contextualization based on a source entity's neighborhood. Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from outputs of the bottom block. We further design a masked entity prediction task to balance information from the relational context and the source entity itself. Experimental results show that HittER achieves new state-of-the-art results on multiple link prediction datasets. We additionally propose a simple approach to integrate HittER into BERT and demonstrate its effectiveness on two Freebase factoid question answering datasets.