SONIC: Spectral Oriented Neural Invariant Convolutions
This work addresses the problem of global context capture and spatial bias in neural networks for computer vision, offering a scalable alternative to conventional methods, though it appears incremental as it builds on prior spectral architectures.
The paper tackled the limitations of CNNs and Vision Transformers in capturing global context and spatial biases by introducing SONIC, a continuous spectral parameterization for convolutional operators, which achieved improved robustness and performance matching or exceeding existing architectures with significantly fewer parameters across various datasets.
Convolutional Neural Networks (CNNs) rely on fixed-size kernels scanning local patches, which limits their ability to capture global context or long-range dependencies without very deep architectures. Vision Transformers (ViTs), in turn, provide global connectivity but lack spatial inductive bias, depend on explicit positional encodings, and remain tied to the initial patch size. Bridging these limitations requires a representation that is both structured and global. We introduce SONIC (Spectral Oriented Neural Invariant Convolutions), a continuous spectral parameterisation that models convolutional operators using a small set of shared, orientation-selective components. These components define smooth responses across the full frequency domain, yielding global receptive fields and filters that adapt naturally across resolutions. Across synthetic benchmarks, large-scale image classification, and 3D medical datasets, SONIC shows improved robustness to geometric transformations, noise, and resolution shifts, and matches or exceeds convolutional, attention-based, and prior spectral architectures with an order of magnitude fewer parameters. These results demonstrate that continuous, orientation-aware spectral parameterisations provide a principled and scalable alternative to conventional spatial and spectral operators.