An MPI-Based Python Framework for Distributed Training with Keras
This is an incremental improvement for researchers and practitioners needing distributed training in supercomputing environments.
The paper tackles the problem of distributed training of neural networks by presenting a lightweight Python framework built on Keras and using MPI for coordination, demonstrating its performance on systems of varying sizes with a benchmark from high-energy physics.
We present a lightweight Python framework for distributed training of neural networks on multiple GPUs or CPUs. The framework is built on the popular Keras machine learning library. The Message Passing Interface (MPI) protocol is used to coordinate the training process, and the system is well suited for job submission at supercomputing sites. We detail the software's features, describe its use, and demonstrate its performance on systems of varying sizes on a benchmark problem drawn from high-energy physics research.