On the Expressive Power of Permutation-Equivariant Weight-Space Networks
This provides a foundational theory for weight-space learning, which is important for researchers and practitioners using pretrained models, though it is incremental as it builds on prior partial results.
The paper tackled the lack of comprehensive theoretical understanding of the expressive power of permutation-equivariant weight-space networks, proving that prominent networks are equivalent in expressive power and establishing universality under mild assumptions while characterizing edge-case failures.
Weight-space learning studies neural architectures that operate directly on the parameters of other neural networks. Motivated by the growing availability of pretrained models, recent work has demonstrated the effectiveness of weight-space networks across a wide range of tasks. SOTA weight-space networks rely on permutation-equivariant designs to improve generalization. However, this may negatively affect expressive power, warranting theoretical investigation. Importantly, unlike other structured domains, weight-space learning targets maps operating on both weight and function spaces, making expressivity analysis particularly subtle. While a few prior works provide partial expressivity results, a comprehensive characterization is still missing. In this work, we address this gap by developing a systematic theory for expressivity of weight-space networks. We first prove that all prominent permutation-equivariant networks are equivalent in expressive power. We then establish universality in both weight- and function-space settings under mild, natural assumptions on the input weights, and characterize the edge-case regimes where universality no longer holds. Together, these results provide a strong and unified foundation for the expressivity of weight-space networks.