Band-limited Training and Inference for Convolutional Neural Networks
This addresses resource efficiency for deep learning practitioners, but it is incremental as it builds on existing CNN compression schemes.
The paper tackles the problem of reducing resource usage in convolutional neural networks (CNNs) by artificially constraining the frequency spectra of filters and data during training, called band-limiting, and finds that CNNs remain resilient with high prediction accuracy while effectively controlling GPU and memory usage without modifying existing algorithms or architectures.
The convolutional layers are core building blocks of neural network architectures. In general, a convolutional filter applies to the entire frequency spectrum of the input data. We explore artificially constraining the frequency spectra of these filters and data, called band-limiting, during training. The frequency domain constraints apply to both the feed-forward and back-propagation steps. Experimentally, we observe that Convolutional Neural Networks (CNNs) are resilient to this compression scheme and results suggest that CNNs learn to leverage lower-frequency components. In particular, we found: (1) band-limited training can effectively control the resource usage (GPU and memory); (2) models trained with band-limited layers retain high prediction accuracy; and (3) requires no modification to existing training algorithms or neural network architectures to use unlike other compression schemes.