Drastically Reducing the Number of Trainable Parameters in Deep CNNs by Inter-layer Kernel-sharing
This addresses memory constraints for edge computing applications, but it is incremental as it builds on existing architectures like ConvMixer and SE-ResNet.
The paper tackles the problem of large memory footprints in deep convolutional neural networks (DCNNs) for edge computing by proposing inter-layer kernel-sharing to reduce trainable parameters, showing on CIFAR-10 and CIFAR-100 that it drastically cuts parameters with minimal accuracy loss.
Deep convolutional neural networks (DCNNs) have become the state-of-the-art (SOTA) approach for many computer vision tasks: image classification, object detection, semantic segmentation, etc. However, most SOTA networks are too large for edge computing. Here, we suggest a simple way to reduce the number of trainable parameters and thus the memory footprint: sharing kernels between multiple convolutional layers. Kernel-sharing is only possible between ``isomorphic" layers, i.e.layers having the same kernel size, input and output channels. This is typically the case inside each stage of a DCNN. Our experiments on CIFAR-10 and CIFAR-100, using the ConvMixer and SE-ResNet architectures show that the number of parameters of these models can drastically be reduced with minimal cost on accuracy. The resulting networks are appealing for certain edge computing applications that are subject to severe memory constraints, and even more interesting if leveraging "frozen weights" hardware accelerators. Kernel-sharing is also an efficient regularization method, which can reduce overfitting. The codes are publicly available at https://github.com/AlirezaAzadbakht/kernel-sharing.