Learning Physical Simulation with Message Passing Transformer
This addresses the challenge of accurate long-term physical simulation for computational physics applications, representing an incremental improvement over existing graph-based methods.
The authors tackled the problem of physical simulation by proposing a Message Passing Transformer architecture with Hadamard-Product Attention and Graph Fourier Loss, achieving significant accuracy improvements in long-term rollouts for Lagrangian and Eulerian dynamical systems.
Machine learning methods for physical simulation have achieved significant success in recent years. We propose a new universal architecture based on Graph Neural Network, the Message Passing Transformer, which incorporates a Message Passing framework, employs an Encoder-Processor-Decoder structure, and applies Graph Fourier Loss as loss function for model optimization. To take advantage of the past message passing state information, we propose Hadamard-Product Attention to update the node attribute in the Processor, Hadamard-Product Attention is a variant of Dot-Product Attention that focuses on more fine-grained semantics and emphasizes on assigning attention weights over each feature dimension rather than each position in the sequence relative to others. We further introduce Graph Fourier Loss (GFL) to balance high-energy and low-energy components. To improve time performance, we precompute the graph's Laplacian eigenvectors before the training process. Our architecture achieves significant accuracy improvements in long-term rollouts for both Lagrangian and Eulerian dynamical systems over current methods.