ANTIC: Adaptive Neural Temporal In-situ Compressor
This addresses a critical bottleneck for high-performance computing infrastructures handling petabyte-to-exabyte scale data from PDE simulations, though it appears incremental as it builds on neural compression methods.
The paper tackles the problem of prohibitive storage requirements for high-resolution spatiotemporal simulations in fields like fluid dynamics and astrophysics by introducing ANTIC, an adaptive neural in situ compressor, achieving storage reductions of several orders of magnitude while maintaining physics accuracy.
The persistent storage requirements for high-resolution, spatiotemporally evolving fields governed by large-scale and high-dimensional partial differential equations (PDEs) have reached the petabyte-to-exabyte scale. Transient simulations modeling Navier-Stokes equations, magnetohydrodynamics, plasma physics, or binary black hole mergers generate data volumes that are prohibitive for modern high-performance computing (HPC) infrastructures. To address this bottleneck, we introduce ANTIC (Adaptive Neural Temporal in situ Compressor), an end-to-end in situ compression pipeline. ANTIC consists of an adaptive temporal selector tailored to high-dimensional physics that identifies and filters informative snapshots at simulation time, combined with a spatial neural compression module based on continual fine-tuning that learns residual updates between adjacent snapshots using neural fields. By operating in a single streaming pass, ANTIC enables a combined compression of temporal and spatial components and effectively alleviates the need for explicit on-disk storage of entire time-evolved trajectories. Experimental results demonstrate how storage reductions of several orders of magnitude relate to physics accuracy.