Edgeformers: Graph-Empowered Transformers for Representation Learning on Textual-Edge Networks
This addresses the challenge of leveraging textual semantics on edges in social and information networks, which is important for applications like user communication analysis, but it is an incremental improvement over existing edge-aware graph neural networks.
The authors tackled the problem of learning representations for networks with text-rich edges by proposing Edgeformers, a Transformer-based framework that contextualizes edge text with network information and aggregates edge representations for nodes, achieving state-of-the-art performance in edge classification and link prediction across five datasets.
Edges in many real-world social/information networks are associated with rich text information (e.g., user-user communications or user-product reviews). However, mainstream network representation learning models focus on propagating and aggregating node attributes, lacking specific designs to utilize text semantics on edges. While there exist edge-aware graph neural networks, they directly initialize edge attributes as a feature vector, which cannot fully capture the contextualized text semantics of edges. In this paper, we propose Edgeformers, a framework built upon graph-enhanced Transformers, to perform edge and node representation learning by modeling texts on edges in a contextualized way. Specifically, in edge representation learning, we inject network information into each Transformer layer when encoding edge texts; in node representation learning, we aggregate edge representations through an attention mechanism within each node's ego-graph. On five public datasets from three different domains, Edgeformers consistently outperform state-of-the-art baselines in edge classification and link prediction, demonstrating the efficacy in learning edge and node representations, respectively.