Efficient compression of neural networks and datasets
This work addresses the challenge of reducing computational and storage costs for neural networks, which is incremental as it builds upon existing compression methods with specific improvements.
The paper tackles the problem of compressing neural networks and datasets by developing methods to reduce parameters while maintaining high test accuracy, resulting in effective data compression algorithms with improved efficiency over previous approaches. It includes comparisons on various architectures and datasets, such as convolutional networks on images and transformers on Wikipedia, and empirically verifies that regularized models can lead to more sample-efficient convergence.
We compare, improve, and contribute methods that substantially decrease the number of parameters of neural networks while maintaining high test accuracy. When applying our methods to minimize description length, we obtain very effective data compression algorithms. In particular, we develop a probabilistic reformulation of $\ell_0$ regularized optimization for nonlinear models that does not require Monte-Carlo sampling and thus improves upon previous methods. We also improve upon methods involving smooth approximations to the $\ell_0$ norm, and investigate layerwise methods. We compare the methods on different architectures and datasets, including convolutional networks trained on image datasets and transformers trained on parts of Wikipedia. We also created a synthetic teacher-student setup to investigate compression in a controlled continuous setting. Finally, we conceptually relate compression algorithms to Solomonoff's theory of inductive inference and empirically verify the prediction that regularized models can exhibit more sample-efficient convergence.