FastPersist: Accelerating Model Checkpointing in Deep Learning
This addresses a critical bottleneck in deep learning training for practitioners, offering a domain-specific optimization that is incremental but impactful.
The paper tackles the problem of slow model checkpointing in deep learning training by proposing FastPersist, which accelerates checkpoint creation up to 116x faster than baseline and enables per-iteration checkpointing with negligible overhead.
Model checkpoints are critical Deep Learning (DL) artifacts that enable fault tolerance for training and downstream applications, such as inference. However, writing checkpoints to persistent storage, and other I/O aspects of DL training, are mostly ignored by compute-focused optimization efforts for faster training of rapidly growing models and datasets. Towards addressing this imbalance, we propose FastPersist to accelerate checkpoint creation in DL training. FastPersist combines three novel techniques: (i) NVMe optimizations for faster checkpoint writes to SSDs, (ii) efficient write parallelism using the available SSDs in training environments, and (iii) overlapping checkpointing with independent training computations. Our evaluation using real world dense and sparse DL models shows that FastPersist creates checkpoints in persistent storage up to 116x faster than baseline, and enables per-iteration checkpointing with negligible overhead.