Asynchronous Local-SGD Training for Language Modeling
This work addresses efficiency in distributed training for language models, offering a practical improvement for researchers and practitioners, though it is incremental as it builds on existing Local-SGD methods.
The paper tackles the challenge of asynchronous Local-SGD for language modeling, where naive implementations lead to slower convergence due to stale gradients, and proposes a method with delayed Nesterov momentum and adaptive local steps that matches synchronous performance in perplexity per update and significantly improves wall clock time, evaluated on models up to 150M parameters.
Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is an approach to distributed optimization where each device performs more than one SGD update per communication. This work presents an empirical study of {\it asynchronous} Local-SGD for training language models; that is, each worker updates the global parameters as soon as it has finished its SGD steps. We conduct a comprehensive investigation by examining how worker hardware heterogeneity, model size, number of workers, and optimizer could impact the learning performance. We find that with naive implementations, asynchronous Local-SGD takes more iterations to converge than its synchronous counterpart despite updating the (global) model parameters more frequently. We identify momentum acceleration on the global parameters when worker gradients are stale as a key challenge. We propose a novel method that utilizes a delayed Nesterov momentum update and adjusts the workers' local training steps based on their computation speed. This approach, evaluated with models up to 150M parameters on the C4 dataset, matches the performance of synchronous Local-SGD in terms of perplexity per update step, and significantly surpasses it in terms of wall clock time.