Impact of Disentanglement on Pruning Neural Networks
This work addresses model compression for edge devices, but it is incremental as it builds on existing VAE and pruning methods.
The study investigated how forcing neural networks to learn disentangled representations affects pruning for classification tasks, finding that disentanglement can improve pruning efficiency on MNIST and CIFAR10 datasets.
Deploying deep learning neural networks on edge devices, to accomplish task specific objectives in the real-world, requires a reduction in their memory footprint, power consumption, and latency. This can be realized via efficient model compression. Disentangled latent representations produced by variational autoencoder (VAE) networks are a promising approach for achieving model compression because they mainly retain task-specific information, discarding useless information for the task at hand. We make use of the Beta-VAE framework combined with a standard criterion for pruning to investigate the impact of forcing the network to learn disentangled representations on the pruning process for the task of classification. In particular, we perform experiments on MNIST and CIFAR10 datasets, examine disentanglement challenges, and propose a path forward for future works.