Equivariant Transformer is all you need
This addresses a bottleneck in computational physics simulations for researchers, though it appears incremental as it combines existing concepts like equivariance and attention.
The paper tackled the problem of poor acceptance rates in self-learning Monte-Carlo (SLMC) by introducing symmetry equivariant attention to impose equivariance, resulting in improved acceptance rates and observation of scaling laws similar to large language models.
Machine learning, deep learning, has been accelerating computational physics, which has been used to simulate systems on a lattice. Equivariance is essential to simulate a physical system because it imposes a strong induction bias for the probability distribution described by a machine learning model. This reduces the risk of erroneous extrapolation that deviates from data symmetries and physical laws. However, imposing symmetry on the model sometimes occur a poor acceptance rate in self-learning Monte-Carlo (SLMC). On the other hand, Attention used in Transformers like GPT realizes a large model capacity. We introduce symmetry equivariant attention to SLMC. To evaluate our architecture, we apply it to our proposed new architecture on a spin-fermion model on a two-dimensional lattice. We find that it overcomes poor acceptance rates for linear models and observe the scaling law of the acceptance rate as in the large language models with Transformers.