Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning
This work addresses the problem of reducing expensive simulation costs for researchers in scientific computing, though it is incremental as it builds on existing neural operator and pretraining techniques.
The authors tackled the data-intensive nature of PDE operator learning by designing unsupervised pretraining with physics-inspired proxy tasks and in-context learning, resulting in a method that is highly data-efficient, more generalizable, and outperforms conventional vision-pretrained models on diverse PDEs.
Recent years have witnessed the promise of coupling machine learning methods and physical domain-specific insights for solving scientific problems based on partial differential equations (PDEs). However, being data-intensive, these methods still require a large amount of PDE data. This reintroduces the need for expensive numerical PDE solutions, partially undermining the original goal of avoiding these expensive simulations. In this work, seeking data efficiency, we design unsupervised pretraining for PDE operator learning. To reduce the need for training data with heavy simulation costs, we mine unlabeled PDE data without simulated solutions, and we pretrain neural operators with physics-inspired reconstruction-based proxy tasks. To improve out-of-distribution performance, we further assist neural operators in flexibly leveraging a similarity-based method that learns in-context examples, without incurring extra training costs or designs. Extensive empirical evaluations on a diverse set of PDEs demonstrate that our method is highly data-efficient, more generalizable, and even outperforms conventional vision-pretrained models. We provide our code at https://github.com/delta-lab-ai/data_efficient_nopt.