The Bearable Lightness of Big Data: Towards Massive Public Datasets in Scientific Machine Learning
This addresses storage constraints for researchers in fields like computational fluid dynamics, though it is incremental as it applies existing compression methods to new data.
The paper tackles the challenge of curating massive high-fidelity simulation datasets for scientific machine learning by showing that deep learning models remain robust to errors from lossy compression, enabling storage-efficient public datasets.
In general, large datasets enable deep learning models to perform with good accuracy and generalizability. However, massive high-fidelity simulation datasets (from molecular chemistry, astrophysics, computational fluid dynamics (CFD), etc. can be challenging to curate due to dimensionality and storage constraints. Lossy compression algorithms can help mitigate limitations from storage, as long as the overall data fidelity is preserved. To illustrate this point, we demonstrate that deep learning models, trained and tested on data from a petascale CFD simulation, are robust to errors introduced during lossy compression in a semantic segmentation problem. Our results demonstrate that lossy compression algorithms offer a realistic pathway for exposing high-fidelity scientific data to open-source data repositories for building community datasets. In this paper, we outline, construct, and evaluate the requirements for establishing a big data framework, demonstrated at https://blastnet.github.io/, for scientific machine learning.