Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling
This work addresses the problem of efficient long-range dependency learning in sequence models for researchers and practitioners, offering an incremental improvement over existing convolution and transformer methods.
The paper tackles the challenge of training long convolutions for sequence modeling by introducing reparameterized multi-resolution convolutions (MRConv), which achieve state-of-the-art performance on benchmarks like Long Range Arena and ImageNet classification.
Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions ($\texttt{MRConv}$), a novel approach to parameterizing global convolutional kernels for long-sequence modelling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, $\texttt{MRConv}$ learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D $\texttt{MRConv}$ layers.