Weight Reparametrization for Budget-Aware Network Pruning
This addresses the cumbersome fine-tuning step in pruning for designing lightweight networks, though it appears incremental as it builds on existing pruning concepts.
The paper tackles the problem of network pruning by introducing an end-to-end method that simultaneously trains and prunes without fine-tuning, achieving competitive results on CIFAR10 and TinyImageNet with standard architectures like Conv4, VGG19, and ResNet18.
Pruning seeks to design lightweight architectures by removing redundant weights in overparameterized networks. Most of the existing techniques first remove structured sub-networks (filters, channels,...) and then fine-tune the resulting networks to maintain a high accuracy. However, removing a whole structure is a strong topological prior and recovering the accuracy, with fine-tuning, is highly cumbersome. In this paper, we introduce an "end-to-end" lightweight network design that achieves training and pruning simultaneously without fine-tuning. The design principle of our method relies on reparametrization that learns not only the weights but also the topological structure of the lightweight sub-network. This reparametrization acts as a prior (or regularizer) that defines pruning masks implicitly from the weights of the underlying network, without increasing the number of training parameters. Sparsity is induced with a budget loss that provides an accurate pruning. Extensive experiments conducted on the CIFAR10 and the TinyImageNet datasets, using standard architectures (namely Conv4, VGG19 and ResNet18), show compelling results without fine-tuning.