State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory
This addresses a fundamental limitation in sequence modeling for researchers and practitioners, but it is incremental as it builds on existing state-space model frameworks.
The paper tackles the limited capacity of state-space models in sequence modeling due to lack of nonlinear activation along the temporal direction, proving that stacking them with layer-wise nonlinear activation enables universal approximation of continuous sequence-to-sequence relationships, while showing theoretically and empirically that these models still suffer from exponential decaying memory.
State-space models have gained popularity in sequence modelling due to their simple and efficient network structures. However, the absence of nonlinear activation along the temporal direction limits the model's capacity. In this paper, we prove that stacking state-space models with layer-wise nonlinear activation is sufficient to approximate any continuous sequence-to-sequence relationship. Our findings demonstrate that the addition of layer-wise nonlinear activation enhances the model's capacity to learn complex sequence patterns. Meanwhile, it can be seen both theoretically and empirically that the state-space models do not fundamentally resolve the issue of exponential decaying memory. Theoretical results are justified by numerical verifications.