Spectral Representations for Convolutional Neural Networks
This work addresses efficiency and optimization challenges in deep learning for researchers and practitioners, though it is incremental as it builds on existing spectral methods.
The authors tackled the problem of improving convolutional neural network (CNN) design and training efficiency by leveraging spectral representations, resulting in innovations like spectral pooling for better information retention and complex-coefficient parameterization for faster convergence. They achieved competitive results on classification and approximation tasks without using dropout or max-pooling.
Discrete Fourier transforms provide a significant speedup in the computation of convolutions in deep learning. In this work, we demonstrate that, beyond its advantages for efficient computation, the spectral domain also provides a powerful representation in which to model and train convolutional neural networks (CNNs). We employ spectral representations to introduce a number of innovations to CNN design. First, we propose spectral pooling, which performs dimensionality reduction by truncating the representation in the frequency domain. This approach preserves considerably more information per parameter than other pooling strategies and enables flexibility in the choice of pooling output dimensionality. This representation also enables a new form of stochastic regularization by randomized modification of resolution. We show that these methods achieve competitive results on classification and approximation tasks, without using any dropout or max-pooling. Finally, we demonstrate the effectiveness of complex-coefficient spectral parameterization of convolutional filters. While this leaves the underlying model unchanged, it results in a representation that greatly facilitates optimization. We observe on a variety of popular CNN configurations that this leads to significantly faster convergence during training.