On the Expressive Power and Limitations of Multi-Layer SSMs
For researchers in sequence modeling, this work provides theoretical insights into the expressive power and limitations of SSMs, clarifying the role of chain-of-thought and resource tradeoffs.
This paper identifies fundamental limitations of multi-layer state-space models (SSMs) in compositional tasks, showing they are less expressive than streaming models. It demonstrates that online chain-of-thought (CoT) can bridge this gap, making SSMs equivalent to streaming algorithms, while offline CoT does not increase expressiveness.
We study the expressive power and limitations of multi-layer state-space models (SSMs). First, we show that multi-layer SSMs face fundamental limitations in compositional tasks, revealing an inherent gap between SSMs and streaming models. Then, we examine the role of chain-of-thought (CoT), showing that offline CoT does not fundamentally increase the expressiveness, while online CoT can substantially increase its power. Indeed, with online CoT, multi-layer SSMs become equivalent in power to streaming algorithms. Finally, we investigate the tradeoff between width and precision, showing that these resources are not interchangeable in the base model, but admit a clean equivalence once online CoT is allowed. Overall, our results offer a unified perspective on how depth, finite precision, and CoT shape the power and limits of SSMs.