Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization
This work addresses the need for efficient spectral normalization in neural networks, particularly for resource-constrained applications, though it is incremental as it builds on existing depthwise separable convolution techniques.
The paper tackled the problem of costly spectral normalization for convolutional layers by introducing a simple method for depthwise separable convolutions that adds negligible computational and memory overhead, demonstrating effectiveness on image classification tasks with MobileNetV2.
An increasing number of models require the control of the spectral norm of convolutional layers of a neural network. While there is an abundance of methods for estimating and enforcing upper bounds on those during training, they are typically costly in either memory or time. In this work, we introduce a very simple method for spectral normalization of depthwise separable convolutions, which introduces negligible computational and memory overhead. We demonstrate the effectiveness of our method on image classification tasks using standard architectures like MobileNetV2.