What's Hidden in a Randomly Weighted Neural Network?
This work addresses the challenge of reducing computational costs in neural network training by revealing that untrained subnetworks can achieve competitive performance, which is incremental but offers practical benefits for efficient model deployment.
The paper tackles the problem of finding high-performing subnetworks within randomly weighted neural networks without training the weights, demonstrating that a subnetwork in a randomly weighted Wide ResNet-50 matches the performance of a trained ResNet-34 on ImageNet.
Training a neural network is synonymous with learning the values of the weights. By contrast, we demonstrate that randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values. Hidden in a randomly weighted Wide ResNet-50 we show that there is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet. Not only do these "untrained subnetworks" exist, but we provide an algorithm to effectively find them. We empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an "untrained subnetwork" approaches a network with learned weights in accuracy. Our code and pretrained models are available at https://github.com/allenai/hidden-networks.