Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention
This work offers theoretical insights for researchers developing sequence models, though it is incremental as it builds on existing Transformer analysis.
This paper provides a theoretical generalization analysis of selective state-space models (SSMs), deriving a covering number-based bound to show how the spectral abscissa of the state matrix affects training stability and generalization across sequence lengths, with empirical validation on synthetic and benchmark tasks.
State-space models (SSMs) have recently emerged as a compelling alternative to Transformers for sequence modeling tasks. This paper presents a theoretical generalization analysis of selective SSMs, the core architectural component behind the Mamba model. We derive a novel covering number-based generalization bound for selective SSMs, building upon recent theoretical advances in the analysis of Transformer models. Using this result, we analyze how the spectral abscissa of the continuous-time state matrix influences the model's stability during training and its ability to generalize across sequence lengths. We empirically validate our findings on a synthetic majority task, the IMDb sentiment classification benchmark, and the ListOps task, demonstrating how our theoretical insights translate into practical model behavior.