Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions
Provides a subquadratic alternative to attention for multi-dimensional data, addressing the need for efficient, structure-preserving operators in domains like genomics, vision, and PDE modeling.
HyenaND introduces a subquadratic, global, input-dependent operator for multi-dimensional data that avoids the spatial structure violations of rasterization. It matches attention baselines in accuracy across genomics, vision, medical imaging, and PDE modeling, with hybrid configurations outperforming both pure attention and recurrence-based hybrids.
Subquadratic alternatives to attention require compromises when applied to multi-dimensional data: standard convolutions lack global receptive fields and input dependency, while recurrent models require rasterizing data such as images, volumes, and partial differential equation (PDE) into an ad-hoc $1\rm D$ scan order that violates their spatial structure. We introduce \textit{HyenaND}, a subquadratic, global, input-dependent operator that acts directly on the native geometry of multidimensional data through convolutions with implicitly parametrized global, input-dependent multi-dimensional convolutional kernels. Our CUDA implementation, \texttt{nSubQ}, fuses the FFT-convolution path to turn HyenaND's $\mathcal{O}(L \log L)$ scaling into wall-clock speedups. Across long-context genomics, computer vision, medical imaging, and PDE modeling, pure HyenaND stacks match the accuracy of strong attention baselines, while hybrid configurations that interleave HyenaND and attention layers outperform both pure attention and strong recurrence-based hybrids.