Symmetry-Aware Fully-Amortized Optimization with Scale Equivariant Graph Metanetworks
This work addresses efficient neural network optimization for machine learning practitioners, but it appears incremental as it builds on existing amortized optimization and symmetry-aware methods.
The paper tackles the problem of accelerating optimization for related tasks by using Scale Equivariant Graph Metanetworks (ScaleGMNs) to enable single-shot fine-tuning, reducing iterative optimization needs, and provides a theoretical insight on scaling symmetries in neural networks.
Amortized optimization accelerates the solution of related optimization problems by learning mappings that exploit shared structure across problem instances. We explore the use of Scale Equivariant Graph Metanetworks (ScaleGMNs) for this purpose. By operating directly in weight space, ScaleGMNs enable single-shot fine-tuning of existing models, reducing the need for iterative optimization. We demonstrate the effectiveness of this approach empirically and provide a theoretical result: the gauge freedom induced by scaling symmetries is strictly smaller in convolutional neural networks than in multi-layer perceptrons. This insight helps explain the performance differences observed between architectures in both our work and that of Kalogeropoulos et al. (2024). Overall, our findings underscore the potential of symmetry-aware metanetworks as a powerful approach for efficient and generalizable neural network optimization. Open-source code: https://github.com/daniuyter/scalegmn_amortization