Compressing multidimensional weather and climate data into neural networks
This enables democratized access to high-resolution climate data for researchers, though it is an incremental improvement in compression methods.
The authors tackled the problem of compressing petabytes of high-resolution weather and climate data by training a coordinate-based neural network to overfit the data, achieving compression ratios from 300x to over 3,000x and outperforming SZ3 in weighted RMSE and MAE. When used as a compressed dataloader for the WeatherBench forecasting model, it increased RMSE by less than 2%.
Weather and climate simulations produce petabytes of high-resolution data that are later analyzed by researchers in order to understand climate change or severe weather. We propose a new method of compressing this multidimensional weather and climate data: a coordinate-based neural network is trained to overfit the data, and the resulting parameters are taken as a compact representation of the original grid-based data. While compression ratios range from 300x to more than 3,000x, our method outperforms the state-of-the-art compressor SZ3 in terms of weighted RMSE, MAE. It can faithfully preserve important large scale atmosphere structures and does not introduce artifacts. When using the resulting neural network as a 790x compressed dataloader to train the WeatherBench forecasting model, its RMSE increases by less than 2%. The three orders of magnitude compression democratizes access to high-resolution climate data and enables numerous new research directions.