Emergence Transformer: Dynamical Temporal Attention Matters
This work addresses the challenge of controlling emergence phenomena in networked dynamics, such as oscillatory coherence in quantum or climate systems, offering a new paradigm for applications like social agreement and machine learning, though it appears incremental as it builds on existing Transformer concepts.
The authors tackled the problem of modulating emergent coherence in complex systems by proposing the Emergence Transformer with dynamical temporal attention (DTA), which reshapes social coherence and enables emergent continual learning in Hopfield networks without catastrophic forgetting.
The Transformer, a breakthrough architecture in artificial intelligence, owes its success to the attention mechanism, which utilizes long-range interactions in sequential data, enabling the emergent coherence between large language models (LLMs) and data distributions. However, temporal attention, that is, different forms of long-range interactions in temporal sequences, has rarely been explored in emergence phenomenon of complex systems including oscillatory coherence in quantum, biophysical, or climate systems. Here, by designing dynamical temporal attention (DTA) with time-varying query, key, and value matrices, we propose an Emergence Transformer. This architecture allows each component to interact with its own or its neighbors' past states through dynamical attention kernels, thereby enabling the promotion and/or suppression of the emergent coherence of components. Interestingly, we uncover that neighbor-DTA consistently promotes oscillatory coherence, whereas self-DTA exhibits an optimal attention weight for coherence enhancement, owing to its non-monotonic dependence on network structure. Practically, we demonstrate how DTA reshapes social coherence, suggesting strategies to either enhance agreement or preserve plurality. We further apply DTA to the paradigmatic Hopfield neural network, achieving emergent continual learning without catastrophic forgetting. Together, these results lay a foundation and provide an immediate paradigm for modulating emergence phenomenon in networked dynamics only using DTA.