Graph Metanetworks for Processing Diverse Neural Architectures
This work addresses the challenge of unifying tasks that treat neural networks as input data, enabling broader applicability across diverse architectures, though it is incremental in building on prior symmetry-aware methods.
The authors tackled the problem of processing diverse neural architectures as input data by developing Graph Metanetworks (GMNs), which represent neural networks as graphs and process them using graph neural networks, achieving generalization to architectures like multi-head attention and normalization layers where previous methods struggled.
Neural networks efficiently encode learned information within their parameters. Consequently, many tasks can be unified by treating neural networks themselves as input data. When doing so, recent studies demonstrated the importance of accounting for the symmetries and geometry of parameter spaces. However, those works developed architectures tailored to specific networks such as MLPs and CNNs without normalization layers, and generalizing such architectures to other types of networks can be challenging. In this work, we overcome these challenges by building new metanetworks - neural networks that take weights from other neural networks as input. Put simply, we carefully build graphs representing the input neural networks and process the graphs using graph neural networks. Our approach, Graph Metanetworks (GMNs), generalizes to neural architectures where competing methods struggle, such as multi-head attention layers, normalization layers, convolutional layers, ResNet blocks, and group-equivariant linear layers. We prove that GMNs are expressive and equivariant to parameter permutation symmetries that leave the input neural network functions unchanged. We validate the effectiveness of our method on several metanetwork tasks over diverse neural network architectures.