Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems
This addresses the issue of long-horizon instability in weather forecasting and similar dynamical systems for researchers and practitioners, representing a novel method for a known bottleneck.
The paper tackles the problem of spectral bias in neural operators for dynamical systems, which attenuates high-frequency components critical for applications like weather forecasting, by proposing multi-scale wavelet transformers (MSWTs) that achieve substantial error reductions and improved long-horizon spectral fidelity, with further reductions in climatological bias on ERA5 climate data.
Recent years have seen a surge in data-driven surrogates for dynamical systems that can be orders of magnitude faster than numerical solvers. However, many machine learning-based models such as neural operators exhibit spectral bias, attenuating high-frequency components that often encode small-scale structure. This limitation is particularly damaging in applications such as weather forecasting, where misrepresented high frequencies can induce long-horizon instability. To address this issue, we propose multi-scale wavelet transformers (MSWTs), which learn system dynamics in a tokenized wavelet domain. The wavelet transform explicitly separates low- and high-frequency content across scales. MSWTs leverage a wavelet-preserving downsampling scheme that retains high-frequency features and employ wavelet-based attention to capture dependencies across scales and frequency bands. Experiments on chaotic dynamical systems show substantial error reductions and improved long horizon spectral fidelity. On the ERA5 climate reanalysis, MSWTs further reduce climatological bias, demonstrating their effectiveness in a real-world forecasting setting.