Continuity and Isolation Lead to Doubts or Dilemmas in Large Language Models
This work addresses a foundational problem for researchers and practitioners in AI and ML by revealing inherent limitations in Transformers, potentially impacting their design and application.
The paper tackles the problem of understanding Transformers' limitations in learning simple pattern sequences by demonstrating and mathematically proving two phenomena, isolation and continuity, which hinder learning, and shows these theoretical limitations occur in practice.
Understanding how Transformers work and how they process information is key to the theoretical and empirical advancement of these machines. In this work, we demonstrate the existence of two phenomena in Transformers, namely isolation and continuity. Both of these phenomena hinder Transformers to learn even simple pattern sequences. Isolation expresses that any learnable sequence must be isolated from another learnable sequence, and hence some sequences cannot be learned by a single Transformer at the same time. Continuity entails that an attractor basin forms around a learned sequence, such that any sequence falling in that basin will collapse towards the learned sequence. Here, we mathematically prove these phenomena emerge in all Transformers that use compact positional encoding, and design rigorous experiments, demonstrating that the theoretical limitations we shed light on occur on the practical scale.