WaLRUS: Wavelets for Long-range Representation Using SSMs
This work addresses a bottleneck in SSMs for researchers and practitioners in sequential data modeling, offering a more flexible approach, though it appears incremental as it builds directly on the existing SaFARi framework.
The paper tackles the limitation of HiPPO-based State-Space Models (SSMs) in modeling long-range dependencies by introducing WaLRUS, a new implementation of the SaFARi framework using Daubechies wavelets, which enables the construction of SSMs from arbitrary frames for greater diversity.
State-Space Models (SSMs) have proven to be powerful tools for modeling long-range dependencies in sequential data. While the recent method known as HiPPO has demonstrated strong performance, and formed the basis for machine learning models S4 and Mamba, it remains limited by its reliance on closed-form solutions for a few specific, well-behaved bases. The SaFARi framework generalized this approach, enabling the construction of SSMs from arbitrary frames, including non-orthogonal and redundant ones, thus allowing an infinite diversity of possible "species" within the SSM family. In this paper, we introduce WaLRUS (Wavelets for Long-range Representation Using SSMs), a new implementation of SaFARi built from Daubechies wavelets.