Masked Autoencoders are PDE Learners
This work addresses the challenge of making neural PDE solvers more practical for physics applications by enhancing their ability to generalize across varied inputs, though it is incremental as it builds on existing masked pretraining methods.
The paper tackles the problem of limited generalizability in neural PDE solvers by adapting masked autoencoders for self-supervised pretraining across diverse PDEs, resulting in learned representations that improve time-stepping and super-resolution performance on unseen equations and parameters.
Neural solvers for partial differential equations (PDEs) have great potential to generate fast and accurate physics solutions, yet their practicality is currently limited by their generalizability. PDEs evolve over broad scales and exhibit diverse behaviors; predicting these phenomena will require learning representations across a wide variety of inputs which may encompass different coefficients, boundary conditions, resolutions, or even equations. As a step towards generalizable PDE modeling, we adapt masked pretraining for physics problems. Through self-supervised learning across PDEs, masked autoencoders can consolidate heterogeneous physics to learn rich latent representations. We show that learned representations can generalize to a limited set of unseen equations or parameters and are meaningful enough to regress PDE coefficients or the classify PDE features. Furthermore, conditioning neural solvers on learned latent representations can improve time-stepping and super-resolution performance across a variety of coefficients, discretizations, or boundary conditions, as well as on certain unseen PDEs. We hope that masked pretraining can emerge as a unifying method across large, unlabeled, and heterogeneous datasets to learn latent physics at scale.