Multi-scale Time-stepping of Partial Differential Equations with Transformers
This work addresses the need for efficient PDE modeling in scientific and engineering applications, representing an incremental improvement over existing transformer-based neural operators.
The paper tackles the problem of developing fast and accurate neural network surrogates for Partial Differential Equations (PDEs) by using a transformer architecture with multi-scale hierarchical time-stepping, achieving similar or better results than state-of-the-art methods like FNO, OFormer, and Galerkin Transformer in predicting Navier-Stokes equations.
Developing fast surrogates for Partial Differential Equations (PDEs) will accelerate design and optimization in almost all scientific and engineering applications. Neural networks have been receiving ever-increasing attention and demonstrated remarkable success in computational modeling of PDEs, however; their prediction accuracy is not at the level of full deployment. In this work, we utilize the transformer architecture, the backbone of numerous state-of-the-art AI models, to learn the dynamics of physical systems as the mixing of spatial patterns learned by a convolutional autoencoder. Moreover, we incorporate the idea of multi-scale hierarchical time-stepping to increase the prediction speed and decrease accumulated error over time. Our model achieves similar or better results in predicting the time-evolution of Navier-Stokes equations compared to the powerful Fourier Neural Operator (FNO) and two transformer-based neural operators OFormer and Galerkin Transformer.