Paraformer: Parameterization of Sub-grid Scale Processes Using Transformers
This work addresses a major source of uncertainty in climate modeling for researchers, though it is incremental by applying an existing attention mechanism to a new domain.
The paper tackles the uncertainty in representing sub-grid scale processes in Global Climate Models by proposing Paraformer, a Transformer-based model trained on the ClimSim dataset, which outperforms classical deep-learning architectures in capturing complex dependencies.
One of the major sources of uncertainty in the current generation of Global Climate Models (GCMs) is the representation of sub-grid scale physical processes. Over the years, a series of deep-learning-based parameterization schemes have been developed and tested on both idealized and real-geography GCMs. However, datasets on which previous deep-learning models were trained either contain limited variables or have low spatial-temporal coverage, which can not fully simulate the parameterization process. Additionally, these schemes rely on classical architectures while the latest attention mechanism used in Transformer models remains unexplored in this field. In this paper, we propose Paraformer, a "memory-aware" Transformer-based model on ClimSim, the largest dataset ever created for climate parameterization. Our results demonstrate that the proposed model successfully captures the complex non-linear dependencies in the sub-grid scale variables and outperforms classical deep-learning architectures. This work highlights the applicability of the attenuation mechanism in this field and provides valuable insights for developing future deep-learning-based climate parameterization schemes.