SD-Conv: Towards the Parameter-Efficiency of Dynamic Convolution
This addresses parameter bloat in efficient CNNs for real-world applications, offering an incremental improvement over existing dynamic convolution methods.
The paper tackles the parameter inefficiency of dynamic convolution in CNNs by proposing SD-Conv, which integrates dynamic mechanisms with sparsity via binary masks, achieving higher performance on ImageNet-1K with reduced parameters and computational cost.
Dynamic convolution achieves better performance for efficient CNNs at the cost of negligible FLOPs increase. However, the performance increase can not match the significantly expanded number of parameters, which is the main bottleneck in real-world applications. Contrastively, mask-based unstructured pruning obtains a lightweight network by removing redundancy in the heavy network. In this paper, we propose a new framework, \textbf{Sparse Dynamic Convolution} (\textsc{SD-Conv}), to naturally integrate these two paths such that it can inherit the advantage of dynamic mechanism and sparsity. We first design a binary mask derived from a learnable threshold to prune static kernels, significantly reducing the parameters and computational cost but achieving higher performance in Imagenet-1K. We further transfer pretrained models into a variety of downstream tasks, showing consistently better results than baselines. We hope our SD-Conv could be an efficient alternative to conventional dynamic convolutions.