Training and Generating Neural Networks in Compressed Weight Space
This work addresses a niche problem for researchers interested in neural network compression and generation, but it is incremental as it aims to open a discussion rather than present a major breakthrough.
The paper tackles the problem of scaling neural networks that process other networks' weight matrices by exploring indirect encodings and compression, specifically using a recurrent neural network to parameterize compressed weights via the discrete cosine transform for character-level language modeling on the enwik8 dataset, but no concrete results or numbers are provided.
The inputs and/or outputs of some neural nets are weight matrices of other neural nets. Indirect encodings or end-to-end compression of weight matrices could help to scale such approaches. Our goal is to open a discussion on this topic, starting with recurrent neural networks for character-level language modelling whose weight matrices are encoded by the discrete cosine transform. Our fast weight version thereof uses a recurrent neural network to parameterise the compressed weights. We present experimental results on the enwik8 dataset.