Making Asynchronous Stochastic Gradient Descent Work for Transformers
This addresses a speed bottleneck for training Transformers in machine translation, though it is incremental as it modifies an existing optimization approach.
The paper tackled the poor convergence of Transformers with asynchronous SGD by proposing a hybrid method that sums several asynchronous updates, achieving near-convergence 1.36x faster in single-node multi-GPU training for neural machine translation without quality loss.
Asynchronous stochastic gradient descent (SGD) is attractive from a speed perspective because workers do not wait for synchronization. However, the Transformer model converges poorly with asynchronous SGD, resulting in substantially lower quality compared to synchronous SGD. To investigate why this is the case, we isolate differences between asynchronous and synchronous methods to investigate batch size and staleness effects. We find that summing several asynchronous updates, rather than applying them immediately, restores convergence behavior. With this hybrid method, Transformer training for neural machine translation task reaches a near-convergence level 1.36x faster in single-node multi-GPU training with no impact on model quality.