W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling
This work addresses the challenge of effective long-range dependency handling in sequence modeling for machine learning applications, offering a scalable foundation for deep SSM-based models, though it is incremental as it builds on existing frameworks like SaFARi and WaLRUS.
The authors tackled the problem of improving long-range sequence modeling in State Space Models (SSMs) by introducing W4S4, a variant based on redundant wavelet frames, which retains information over long horizons significantly better than HiPPO-based SSMs, as shown in experiments across delay reconstruction, classification, and sequence modeling tasks.
State Space Models (SSMs) have emerged as powerful components for sequence modeling, enabling efficient handling of long-range dependencies via linear recurrence and convolutional computation. However, their effectiveness depends heavily on the choice and initialization of the state matrix. In this work, we build on the SaFARi framework and existing WaLRUS SSMs to introduce a new variant, W4S4 (WaLRUS for S4), a new class of SSMs constructed from redundant wavelet frames. WaLRUS admits a stable diagonalization and supports fast kernel computation without requiring low-rank approximations, making it both theoretically grounded and computationally efficient. We show that WaLRUS retains information over long horizons significantly better than HiPPO-based SSMs, both in isolation and when integrated into deep architectures such as S4. Our experiments demonstrate consistent improvements across delay reconstruction tasks, classification benchmarks, and long-range sequence modeling, confirming that high-quality, structured initialization enabled by wavelet-based state dynamic offers substantial advantages over existing alternatives. WaLRUS provides a scalable and versatile foundation for the next generation of deep SSM-based models.