Spectral Convolution Networks
This work addresses a bottleneck for researchers and practitioners in deep learning by enabling more efficient spectral implementations of convolution networks, though it is incremental as it builds on existing spectral methods.
The paper tackles the computational inefficiency of repeatedly transforming between spatial and frequency domains in spectral convolution networks by mathematically deriving how both convolution and activation can be implemented directly in the frequency domain using Fourier or Laplace transforms, resulting in reduced transform requirements and overall complexity.
Previous research has shown that computation of convolution in the frequency domain provides a significant speedup versus traditional convolution network implementations. However, this performance increase comes at the expense of repeatedly computing the transform and its inverse in order to apply other network operations such as activation, pooling, and dropout. We show, mathematically, how convolution and activation can both be implemented in the frequency domain using either the Fourier or Laplace transformation. The main contributions are a description of spectral activation under the Fourier transform and a further description of an efficient algorithm for computing both convolution and activation under the Laplace transform. By computing both the convolution and activation functions in the frequency domain, we can reduce the number of transforms required, as well as reducing overall complexity. Our description of a spectral activation function, together with existing spectral analogs of other network functions may then be used to compose a fully spectral implementation of a convolution network.