Can Language Models Capture Graph Semantics? From Graphs to Language Model and Vice-Versa
This highlights a fundamental limitation in applying language models to graph-structured data, which is crucial for AI tasks involving semantic knowledge representation.
The study investigated whether Transformer models can compress and reconstruct knowledge graphs while preserving semantics, finding that they fail to fully capture the directed, relationship-based structure of graphs due to mismatches with the undirected attention mechanism.
Knowledge Graphs are a great resource to capture semantic knowledge in terms of entities and relationships between the entities. However, current deep learning models takes as input distributed representations or vectors. Thus, the graph is compressed in a vectorized representation. We conduct a study to examine if the deep learning model can compress a graph and then output the same graph with most of the semantics intact. Our experiments show that Transformer models are not able to express the full semantics of the input knowledge graph. We find that this is due to the disparity between the directed, relationship and type based information contained in a Knowledge Graph and the fully connected token-token undirected graphical interpretation of the Transformer Attention matrix.