Selection Mechanisms for Sequence Modeling using Linear State Space Models
This work offers an incremental improvement for researchers in machine learning and control theory by providing a complementary perspective on selection mechanisms in sequence modeling.
The paper tackles the problem of improving selection mechanisms in sequence modeling by proposing a novel residual generator inspired by control theory, which combines multiple Linear Time-Invariant systems to achieve selectivity comparable to existing methods like Mamba, as tested on synthetic benchmarks.
Recent advancements in language modeling tasks have been driven by architectures such as Transformers and, more recently, by Selective State Space Models (SSMs). In this paper, we introduce an alternative selection mechanism inspired by control theory methodologies. Specifically, we propose a novel residual generator for selection, drawing an analogy to fault detection strategies in Linear Time-Invariant (LTI) systems. Unlike Mamba, which utilizes Linear Time-Varying (LTV) systems, our approach combines multiple LTI systems, preserving their beneficial properties during training while achieving comparable selectivity. To evaluate the effectiveness of the proposed architecture, we test its performance on synthetic tasks. While these tasks are not inherently critical, they serve as benchmarks to test the selectivity properties of different cores architecture. This work highlights the potential of integrating theoretical insights with experimental advancements, offering a complementary perspective to deep learning innovations at the intersection of control theory and machine learning.