FALCON: Lightweight and Accurate Convolution
This work addresses the need for lightweight CNN models for deployment in resource-constrained environments, offering an incremental improvement over existing depthwise separable convolution methods.
The paper tackles the problem of compressing Convolutional Neural Networks (CNNs) efficiently while retaining accuracy by proposing FALCON, a method that uses a mathematical formulation (GEP) to approximate standard convolution kernels, achieving up to 8x reduction in parameters and FLOPs with similar accuracy.
How can we efficiently compress Convolutional Neural Network (CNN) while retaining their accuracy on classification tasks? Depthwise Separable Convolution (DSConv), which replaces a standard convolution with a depthwise convolution and a pointwise convolution, has been used for building lightweight architectures. However, previous works based on depthwise separable convolution are limited when compressing a trained CNN model since 1) they are mostly heuristic approaches without a precise understanding of their relations to standard convolution, and 2) their accuracies do not match that of the standard convolution. In this paper, we propose FALCON, an accurate and lightweight method to compress CNN. FALCON uses GEP, our proposed mathematical formulation to approximate the standard convolution kernel, to interpret existing convolution methods based on depthwise separable convolution. By exploiting the knowledge of a trained standard model and carefully determining the order of depthwise separable convolution via GEP, FALCON achieves sufficient accuracy close to that of the trained standard model. Furthermore, this interpretation leads to developing a generalized version rank-k FALCON which performs k independent FALCON operations and sums up the result. Experiments show that FALCON 1) provides higher accuracy than existing methods based on depthwise separable convolution and tensor decomposition, and 2) reduces the number of parameters and FLOPs of standard convolution by up to a factor of 8 while ensuring similar accuracy. We also demonstrate that rank-k FALCON further improves the accuracy while sacrificing a bit of compression and computation reduction rates.