On the Internal Representations of Graph Metanetworks
This work addresses a fundamental gap in understanding internal representations for researchers in deep learning, though it is incremental as it builds on existing GMN methods.
The paper tackled the problem of understanding how graph metanetworks (GMNs) learn from parameters in weight space learning, revealing through experiments that GMNs differ from general neural networks like MLPs and CNNs in their representation space.
Weight space learning is an emerging paradigm in the deep learning community. The primary goal of weight space learning is to extract informative features from a set of parameters using specially designed neural networks, often referred to as \emph{metanetworks}. However, it remains unclear how these metanetworks learn solely from parameters. To address this, we take the first step toward understanding \emph{representations} of metanetworks, specifically graph metanetworks (GMNs), which achieve state-of-the-art results in this field, using centered kernel alignment (CKA). Through various experiments, we reveal that GMNs and general neural networks (\textit{e.g.,} multi-layer perceptrons (MLPs) and convolutional neural networks (CNNs)) differ in terms of their representation space.