Quantitative Clustering in Mean-Field Transformer Models
This work provides theoretical insights into the synchronization behavior of transformer models, which is incremental as it builds on prior mean-field analyses.
The paper tackles the problem of understanding long-term token clustering in mean-field transformer models, establishing exponential contraction rates to a Dirac point mass for regular initializations under certain parameter assumptions.
The evolution of tokens through a deep transformer models can be modeled as an interacting particle system that has been shown to exhibit an asymptotic clustering behavior akin to the synchronization phenomenon in Kuramoto models. In this work, we investigate the long-time clustering of mean-field transformer models. More precisely, we establish exponential rates of contraction to a Dirac point mass for any suitably regular initialization under some assumptions on the parameters of transformer models, any suitably regular mean-field initialization synchronizes exponentially fast with some quantitative rates.