Design Principles for Sequence Models via Coefficient Dynamics
This work provides foundational insights for researchers in machine learning, particularly those designing sequence models, by explaining empirical successes and offering systematic design principles, though it is incremental in synthesizing existing literature.
The authors tackled the problem of understanding and comparing diverse deep sequence models by developing a unified framework that casts output coefficients as outputs of autonomous linear dynamical systems, revealing common mathematical themes and deriving design principles linking architectural choices to model properties.
Deep sequence models, ranging from Transformers and State Space Models (SSMs) to more recent approaches such as gated linear RNNs, fundamentally compute outputs as linear combinations of past value vectors. To draw insights and systematically compare such architectures, we develop a unified framework that makes this output operation explicit, by casting the linear combination coefficients as the outputs of autonomous linear dynamical systems driven by impulse inputs. This viewpoint, in spirit substantially different from approaches focusing on connecting linear RNNs with linear attention, reveals a common mathematical theme across diverse architectures and crucially captures softmax attention, on top of RNNs, SSMs, and related models. In contrast to new model proposals that are commonly evaluated on benchmarks, we derive design principles linking architectural choices to model properties. Thereby identifying tradeoffs between expressivity and efficient implementation, geometric constraints on input selectivity, and stability conditions for numerically stable training and information retention. By connecting several insights and observations from recent literature, the framework both explains empirical successes of recent designs and provides guiding principles for systematically designing new sequence model architectures.