Generating Neural Networks with Neural Networks
This work addresses the challenge of weight generation for neural networks, which is incremental as it builds on existing hypernetwork concepts with a new training formulation.
The paper tackles the problem of generating diverse neural network weights via hypernetworks by formulating a training objective that balances accuracy and diversity, considering trivial symmetry transformations, and demonstrates that the generated weights are diverse and lie on a non-trivial manifold.
Hypernetworks are neural networks that generate weights for another neural network. We formulate the hypernetwork training objective as a compromise between accuracy and diversity, where the diversity takes into account trivial symmetry transformations of the target network. We explain how this simple formulation generalizes variational inference. We use multi-layered perceptrons to form the mapping from the low dimensional input random vector to the high dimensional weight space, and demonstrate how to reduce the number of parameters in this mapping by parameter sharing. We perform experiments and show that the generated weights are diverse and lie on a non-trivial manifold.