Orbax: Distributed Checkpointing with JAX
For JAX users in distributed ML, Orbax provides a modular and efficient checkpointing solution, though it is incremental as it addresses a known gap.
Orbax is a JAX-native checkpointing library that outperforms PyTorch competitors by up to 3.5× for saving and 2× for loading, addressing the lack of a standardized checkpointing solution in JAX.
In a landscape of high-performance distributed ML systems, JAX has emerged as a framework of choice. However, JAX's modular design philosophy leaves it without a standardized checkpointing solution. In this paper, we introduce Orbax, a modular, JAX-native checkpointing library that abstracts the complexities of distributed accelerator systems while also providing flexibility for user-friendly checkpoint manipulations throughout the ML model lifecycle. We demonstrate performance exceeding comparable PyTorch competitors by up to 3.5$\times$ for saving and 2$\times$ for loading. The library is available at https://github.com/google/orbax.