Spiking Sequence Machines and Transformers
For sequence learning researchers, this work unifies two seemingly disparate architectures under a common framework, but the findings are primarily theoretical and incremental.
The paper reveals that spiking Sparse Distributed Memory and transformers share five functional operations with cosine similarity as the retrieval primitive, and proves a Phase-Latency Isomorphism linking sinusoidal phase and spike timing. It shows that distance discriminability under dot-product similarity, not sinusoidal form, is critical for positional encoding.
Sequence learning reduces to similarity-based retrieval over a temporally indexed representation space, a constraint on any sequence model, not a property of a specific architecture. We show that a spiking Sparse Distributed Memory sequence machine (2007) and the transformer (2017) independently instantiate the same five functional operations (encoding, context maintenance, associative retrieval, storage, and decoding), with cosine similarity as the shared retrieval primitive in both. We formalise a Phase-Latency Isomorphism showing that sinusoidal positional phase and spike timing are linearly related, and prove that dot product attention is invariant to this mapping up to a global scale factor on the positional component (Lemma 1). Empirically, frequency-compressed positional encoding fails to converge on a positionally demanding copy task, while a learned rank-based embedding matches or exceeds sinusoidal encoding, indicating that the critical property for positional representation is distance discriminability under dot-product similarity, not sinusoidal form. Time, phase, and rank are three instantiations of the same computational primitive, an ordered index whose structure survives similarity-based retrieval.