Transformer models are gauge invariant: A mathematical connection between AI and particle physics
This work connects AI and physics by identifying a mathematical symmetry in transformers, which is incremental as it applies an existing concept to a new domain.
The paper demonstrates that transformer architectures exhibit gauge invariance, a symmetry property from particle physics, and shows that the default representation partially but not fully removes this invariance.
In particle physics, the fundamental forces are subject to symmetries called gauge invariance. It is a redundancy in the mathematical description of any physical system. In this article I will demonstrate that the transformer architecture exhibits the same properties, and show that the default representation of transformers has partially, but not fully removed the gauge invariance.