Compact Circulant Layers with Spectral Priors
This addresses the need for memory-efficient, uncertainty-aware neural networks in critical applications like medicine and robotics, though it appears incremental as it builds on existing circulant layer concepts with spectral enhancements.
The paper tackles the problem of creating compact, uncertainty-aware neural networks for resource-constrained deployments by introducing spectral circulant and BCCB layers that parameterize filters in the frequency domain. The result shows these layers match strong baselines on tasks like MNIST->Fashion-MNIST and CIFAR-10/Tiny ImageNet while using substantially fewer parameters and providing tighter Lipschitz certificates.
Critical applications in areas such as medicine, robotics and autonomous systems require compact (i.e., memory efficient), uncertainty-aware neural networks suitable for edge and other resource-constrained deployments. We study compact spectral circulant and block-circulant-with-circulant-blocks (BCCB) layers: FFT-diagonalizable circular convolutions whose weights live directly in the real FFT (RFFT) half (1D) or half-plane (2D). Parameterizing filters in the frequency domain lets us impose simple spectral structure, perform structured variational inference in a low-dimensional weight space, and calculate exact layer spectral norms, enabling inexpensive global Lipschitz bounds and margin-based robustness diagnostics. By placing independent complex Gaussians on the Hermitian support we obtain a discrete instance of the spectral representation of stationary kernels, inducing an exact stationary Gaussian-process prior over filters on the discrete circle/torus. We exploit this to define a practical spectral prior and a Hermitian-aware low-rank-plus-diagonal variational posterior in real coordinates. Empirically, spectral circulant/BCCB layers are effective compact building blocks in both (variational) Bayesian and point estimate regimes: compact Bayesian neural networks on MNIST->Fashion-MNIST, variational heads on frozen CIFAR-10 features, and deterministic ViT projections on CIFAR-10/Tiny ImageNet; spectral layers match strong baselines while using substantially fewer parameters and with tighter Lipschitz certificates.