Distributed Training and Optimization Of Neural Networks
This paper addresses the problem of high computational resource requirements and long training times for deep learning models, which is a significant challenge for researchers in high energy physics.
This paper reviews methods for distributed training and optimization of neural networks, addressing the computational challenges posed by large models and datasets. The authors discuss various parallel computation techniques within the context of high energy physics.
Deep learning models are yielding increasingly better performances thanks to multiple factors. To be successful, model may have large number of parameters or complex architectures and be trained on large dataset. This leads to large requirements on computing resource and turn around time, even more so when hyper-parameter optimization is done (e.g search over model architectures). While this is a challenge that goes beyond particle physics, we review the various ways to do the necessary computations in parallel, and put it in the context of high energy physics.