A Deeper Look into Convolutions via Eigenvalue-based Pruning
This work aims to improve the efficiency of convolutional neural networks for practitioners by providing a new method to identify and remove redundant kernels, which is an incremental improvement over existing pruning techniques.
This paper addresses the problem of identifying and pruning redundant convolutional kernels in ResNet architectures without performance loss. It proposes an eigenvalue-based pruning method, contrasting it with the standard average absolute weight approach, and evaluates it on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets.
Convolutional neural networks (CNNs) are able to attain better visual recognition performance than fully connected neural networks despite having much fewer parameters due to their parameter sharing principle. Modern architectures usually contain a small number of fully-connected layers, often at the end, after multiple layers of convolutions. In some cases, most of the convolutions can be eliminated without suffering any loss in recognition performance. However, there is no solid recipe to detect the hidden subset of convolutional neurons that is responsible for the majority of the recognition work. In this work, we formulate this as a pruning problem where the aim is to prune as many kernels as possible while preserving the vanilla generalization performance. To this end, we use the matrix characteristics based on eigenvalues for pruning, in comparison to the average absolute weight of a kernel which is the de facto standard in the literature to assess the importance of an individual convolutional kernel, to shed light on the internal mechanisms of a widely used family of CNNs, namely residual neural networks (ResNets), for the image classification problem using CIFAR-10, CIFAR-100 and Tiny ImageNet datasets.