Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalization
This work addresses model merging challenges for neural network practitioners, though it builds incrementally on prior symmetry-handling approaches.
The paper tackles the problem of neural network symmetries creating multiple equivalent minima in the loss landscape, which complicates model merging. By incorporating both permutation and scaling symmetries into an autoencoder framework, the method enables smooth linear interpolation between models like INRs and CNNs without solving computationally intensive assignment problems.
Neural network parameterizations exhibit inherent symmetries that yield multiple equivalent minima within the loss landscape. Scale Graph Metanetworks (ScaleGMNs) explicitly leverage these symmetries by proposing an architecture equivariant to both permutation and parameter scaling transformations. Previous work by Ainsworth et al. (2023) addressed permutation symmetries through a computationally intensive combinatorial assignment problem, demonstrating that leveraging permutation symmetries alone can map networks into a shared loss basin. In this work, we extend their approach by also incorporating scaling symmetries, presenting an autoencoder framework utilizing ScaleGMNs as invariant encoders. Experimental results demonstrate that our method aligns Implicit Neural Representations (INRs) and Convolutional Neural Networks (CNNs) under both permutation and scaling symmetries without explicitly solving the assignment problem. This approach ensures that similar networks naturally converge within the same basin, facilitating model merging, i.e., smooth linear interpolation while avoiding regions of high loss. The code is publicly available on our GitHub repository.