Probing Graph Representations
This work addresses the need for better diagnostic tools to evaluate graph-based models, particularly for researchers in graph machine learning, though it is incremental as it applies existing probing techniques to a new domain.
The paper tackled the problem of understanding what information is encoded in learned graph representations by using a probing framework to quantify meaningful information, finding on molecular datasets that transformer-based models capture more chemically relevant information than message-passing models.
Today we have a good theoretical understanding of the representational power of Graph Neural Networks (GNNs). For example, their limitations have been characterized in relation to a hierarchy of Weisfeiler-Lehman (WL) isomorphism tests. However, we do not know what is encoded in the learned representations. This is our main question. We answer it using a probing framework to quantify the amount of meaningful information captured in graph representations. Our findings on molecular datasets show the potential of probing for understanding the inductive biases of graph-based models. We compare different families of models and show that transformer-based models capture more chemically relevant information compared to models based on message passing. We also study the effect of different design choices such as skip connections and virtual nodes. We advocate for probing as a useful diagnostic tool for evaluating graph-based models.