Real-time Inference and Extrapolation via a Diffusion-inspired Temporal Transformer Operator (DiTTO)
This addresses the problem of extrapolation in PDE modeling for domains like climate science and fluid dynamics, offering a novel method that is incremental in combining diffusion models and transformers.
The authors tackled the challenge of extrapolation in deep neural networks for time-dependent PDEs by proposing DiTTO, an operator learning method that enables real-time inference and extrapolation without temporal discretization, demonstrating performance on climate and hypersonic flow problems with capabilities like long-time extrapolation and zero-shot super-resolution.
Extrapolation remains a grand challenge in deep neural networks across all application domains. We propose an operator learning method to solve time-dependent partial differential equations (PDEs) continuously and with extrapolation in time without any temporal discretization. The proposed method, named Diffusion-inspired Temporal Transformer Operator (DiTTO), is inspired by latent diffusion models and their conditioning mechanism, which we use to incorporate the temporal evolution of the PDE, in combination with elements from the transformer architecture to improve its capabilities. Upon training, DiTTO can make inferences in real-time. We demonstrate its extrapolation capability on a climate problem by estimating the temperature around the globe for several years, and also in modeling hypersonic flows around a double-cone. We propose different training strategies involving temporal-bundling and sub-sampling and demonstrate performance improvements for several benchmarks, performing extrapolation for long time intervals as well as zero-shot super-resolution in time.