Recurrent Convolution for Compact and Cost-Adjustable Neural Networks: An Empirical Study
This work addresses model efficiency for deep learning practitioners, though it appears incremental as it builds on existing recurrent convolution concepts.
The paper tackles the problem of model compression in convolutional neural networks by proposing a variant of recurrent convolution with independently learned batch normalization layers for different unrolling steps, achieving competitive performance on image classification and denoising tasks.
Recurrent convolution (RC) shares the same convolutional kernels and unrolls them multiple steps, which is originally proposed to model time-space signals. We argue that RC can be viewed as a model compression strategy for deep convolutional neural networks. RC reduces the redundancy across layers. However, the performance of an RC network is not satisfactory if we directly unroll the same kernels multiple steps. We propose a simple yet effective variant which improves the RC networks: the batch normalization layers of an RC module are learned independently (not shared) for different unrolling steps. Moreover, we verify that RC can perform cost-adjustable inference which is achieved by varying its unrolling steps. We learn double independent BN layers for cost-adjustable RC networks, i.e. independent w.r.t both the unrolling steps of current cell and upstream cell. We provide insights on why the proposed method works successfully. Experiments on both image classification and image denoise demonstrate the effectiveness of our method.