Tadpole: Autoencoders as Foundation Models for 3D PDEs with Online Learning
For researchers in scientific machine learning, Tadpole provides a versatile and scalable foundation model for 3D PDEs, though it is an incremental step building on existing autoencoder and fine-tuning techniques.
Tadpole is a foundation model for 3D PDEs pre-trained as an autoencoder on synthetic data generated by an online framework, scaling to hundreds of terabytes. It achieves strong fine-tuning performance on downstream tasks like dynamics learning and generative modeling with minimal trainable parameters.
We introduce Tadpole, a novel foundation model for three-dimensional partial differential equations (PDEs) that addresses key challenges in transferability, scalability to high dimensionality, and multi-functionality. Tadpole is pre-trained as an autoencoder on synthetic 3D PDE data generated by an efficient online data-generation framework. This enables large-scale, diverse training without storage or I/O overhead, demonstrated by scaling to an equivalent of hundreds of terabytes of training data. By autoencoding single-channel spatial crops, Tadpole learns rich and transferable representations across heterogeneous physical systems with varying numbers of state variables and spatial resolutions. Although pre-trained solely as an autoencoder, Tadpole can be efficiently applied for multiple downstream tasks beyond reconstruction, including dynamics learning and generative modeling. For dynamics learning, we propose a novel parameter-efficient fine-tuning strategy that integrates low-rank adaptation, latent-space transformations, and reintroduced skip connections, achieving accurate temporal modeling with a minimal number of trainable parameters. Tadpole demonstrates strong fine-tuning performance across various downstream tasks, highlighting its versatility and effectiveness as a foundation model for 3D PDE learning. Source code and pre-trained weights of Tadpole are available at https://github.com/tum-pbs/tadpole