DeepSquare: Boosting the Learning Power of Deep Convolutional Neural Networks with Elementwise Square Operators
This work addresses the need for efficient neural network modules to boost performance in resource-constrained environments like mobile models, though it is incremental as it builds on existing architectures.
The paper tackles the problem of enhancing deep convolutional neural networks' learning power without adding significant computational cost by introducing lightweight modules based on elementwise square operators, achieving a 1.45% absolute improvement in top-1 accuracy on ImageNet 2012 with ShuffleNetV2-0.5x and no extra parameters.
Modern neural network modules which can significantly enhance the learning power usually add too much computational complexity to the original neural networks. In this paper, we pursue very efficient neural network modules which can significantly boost the learning power of deep convolutional neural networks with negligible extra computational cost. We first present both theoretically and experimentally that elementwise square operator has a potential to enhance the learning power of neural networks. Then, we design four types of lightweight modules with elementwise square operators, named as Square-Pooling, Square-Softmin, Square-Excitation, and Square-Encoding. We add our four lightweight modules to Resnet18, Resnet50, and ShuffleNetV2 for better performance in the experiment on ImageNet 2012 dataset. The experimental results show that our modules can bring significant accuracy improvements to the base convolutional neural network models. The performance of our lightweight modules is even comparable to many complicated modules such as bilinear pooling, Squeeze-and-Excitation, and Gather-Excite. Our highly efficient modules are particularly suitable for mobile models. For example, when equipped with a single Square-Pooling module, the top-1 classification accuracy of ShuffleNetV2-0.5x on ImageNet 2012 is absolutely improved by 1.45% with no additional parameters and negligible inference time overhead.