Age-Based Coded Computation for Bias Reduction in Distributed Learning
This addresses convergence issues in distributed learning systems with correlated stragglers, though it appears incremental as an enhancement to existing coded computation methods.
The paper tackles biased gradient estimators in distributed learning with straggling workers by proposing a timely dynamic encoding framework with an age metric to regulate recovery frequencies, showing through numerical results that it reduces bias and accelerates convergence compared to static schemes.
Coded computation can be used to speed up distributed learning in the presence of straggling workers. Partial recovery of the gradient vector can further reduce the computation time at each iteration; however, this can result in biased estimators, which may slow down convergence, or even cause divergence. Estimator bias will be particularly prevalent when the straggling behavior is correlated over time, which results in the gradient estimators being dominated by a few fast servers. To mitigate biased estimators, we design a $timely$ dynamic encoding framework for partial recovery that includes an ordering operator that changes the codewords and computation orders at workers over time. To regulate the recovery frequencies, we adopt an $age$ metric in the design of the dynamic encoding scheme. We show through numerical results that the proposed dynamic encoding strategy increases the timeliness of the recovered computations, which as a result, reduces the bias in model updates, and accelerates the convergence compared to the conventional static partial recovery schemes.