Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling
This addresses the computational bottleneck in sequence modeling for AI applications, offering a more efficient alternative to existing models.
The paper tackles the problem of inefficient sequence modeling by introducing LrcSSM, a non-linear recurrent model that processes long sequences as fast as linear state-space layers while providing formal gradient-stability guarantees. It demonstrates superior performance over Transformers, LRU, S5, and Mamba on long-range forecasting tasks.
We present LrcSSM, a $\textit{non-linear}$ recurrent model that processes long sequences as fast as today's linear state-space layers. By forcing its Jacobian matrix to be diagonal, the full sequence can be solved in parallel, giving $\mathcal{O}(TD)$ time and memory and only $\mathcal{O}(\log T)$ sequential depth, for input-sequence length $T$ and a state dimension $D$. Moreover, LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 and Mamba do not provide. Importantly, the diagonal Jacobian structure of our model results in no performance loss compared to the original model with dense Jacobian, and the approach can be generalized to other non-linear recurrent models, demonstrating broader applicability. On a suite of long-range forecasting tasks, we demonstrate that LrcSSM outperforms Transformers, LRU, S5, and Mamba.