Renormalized Sparse Neural Network Pruning
This addresses the issue of maintaining accuracy in pruned neural networks for machine learning practitioners, offering a novel theoretical and practical solution.
The paper tackles the problem of accuracy loss in sparse neural networks after pruning by proposing renormalization, proving that it enables error convergence to zero and showing large accuracy improvements on datasets like MNIST, Fashion MNIST, and CIFAR-10.
Large neural networks are heavily over-parameterized. This is done because it improves training to optimality. However once the network is trained, this means many parameters can be zeroed, or pruned, leaving an equivalent sparse neural network. We propose renormalizing sparse neural networks in order to improve accuracy. We prove that our method's error converges to zero as network parameters cluster or concentrate. We prove that without renormalizing, the error does not converge to zero in general. We experiment with our method on real world datasets MNIST, Fashion MNIST, and CIFAR-10 and confirm a large improvement in accuracy with renormalization versus standard pruning.