Superposition in Graph Neural Networks
This work addresses interpretability challenges for researchers and practitioners using GNNs, though it is incremental as it builds on existing methods with controlled experiments.
The authors tackled the difficulty of interpreting graph neural networks (GNNs) by studying superposition in their latent spaces, finding that design choices like width, pooling, and activations affect feature overlap and alignment, which can guide more interpretable GNN development.
Interpreting graph neural networks (GNNs) is difficult because message passing mixes signals and internal channels rarely align with human concepts. We study superposition, the sharing of directions by multiple features, directly in the latent space of GNNs. Using controlled experiments with unambiguous graph concepts, we extract features as (i) class-conditional centroids at the graph level and (ii) linear-probe directions at the node level, and then analyze their geometry with simple basis-invariant diagnostics. Across GCN/GIN/GAT we find: increasing width produces a phase pattern in overlap; topology imprints overlap onto node-level features that pooling partially remixes into task-aligned graph axes; sharper pooling increases axis alignment and reduces channel sharing; and shallow models can settle into metastable low-rank embeddings. These results connect representational geometry with concrete design choices (width, pooling, and final-layer activations) and suggest practical approaches for more interpretable GNNs.