UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation
This addresses the data- and compute-intensive challenge of building foundation models for PDE solving across domains, though it is incremental as it builds on existing neural operator and LLM adaptation techniques.
The paper tackles the problem of developing unified neural operators for diverse PDE families by proposing UPS, which embeds different PDEs into a shared representation space and warm-starts from pretrained LLMs with explicit alignment. The method achieves state-of-the-art results on PDEBench using 4 times less data and 26 times less compute than existing unified models.
We present Unified PDE Solvers (UPS), a data- and compute-efficient approach to developing unified neural operators for diverse families of spatiotemporal PDEs from various domains, dimensions, and resolutions. UPS embeds different PDEs into a shared representation space and processes them using a FNO-transformer architecture. Rather than training the network from scratch, which is data-demanding and computationally expensive, we warm-start the transformer from pretrained LLMs and perform explicit alignment to reduce the modality gap while improving data and compute efficiency. The cross-modal UPS achieves state-of-the-art results on a wide range of 1D and 2D PDE families from PDEBench, outperforming existing unified models using 4 times less data and 26 times less compute. Meanwhile, it is capable of few-shot transfer to unseen PDE families and coefficients.