On the Benefits of Memory for Modeling Time-Dependent PDEs
This work addresses the challenge of improving accuracy for time-dependent PDE modeling in computational science, particularly for noisy or low-resolution data, though it is incremental as it builds on existing neural operator architectures.
The paper tackles the problem of modeling time-dependent PDEs by demonstrating that using memory (past states) can be arbitrarily better than Markovian approaches, and introduces MemNO, which achieves up to 6x reduction in test error, especially in low-resolution or noisy scenarios.
Data-driven techniques have emerged as a promising alternative to traditional numerical methods for solving PDEs. For time-dependent PDEs, many approaches are Markovian -- the evolution of the trained system only depends on the current state, and not the past states. In this work, we investigate the benefits of using memory for modeling time-dependent PDEs: that is, when past states are explicitly used to predict the future. Motivated by the Mori-Zwanzig theory of model reduction, we theoretically exhibit examples of simple (even linear) PDEs, in which a solution that uses memory is arbitrarily better than a Markovian solution. Additionally, we introduce Memory Neural Operator (MemNO), a neural operator architecture that combines recent state space models (specifically, S4) and Fourier Neural Operators (FNOs) to effectively model memory. We empirically demonstrate that when the PDEs are supplied in low resolution or contain observation noise at train and test time, MemNO significantly outperforms the baselines without memory -- with up to 6x reduction in test error. Furthermore, we show that this benefit is particularly pronounced when the PDE solutions have significant high-frequency Fourier modes (e.g., low-viscosity fluid dynamics) and we construct a challenging benchmark dataset consisting of such PDEs.