Flag Aggregator: Scalable Distributed Training under Failures and Augmented Losses using Convex Optimization
This addresses the need for scalable and reliable distributed training systems for large-scale ML applications, representing an incremental improvement over existing Byzantine resilient methods.
The paper tackles the problem of robust aggregation in distributed training under Byzantine failures and data augmentation by formulating it as a Maximum Likelihood Estimation using Beta densities, showing that their approach significantly enhances robustness and improves communication efficiency and accuracy in various tasks.
Modern ML applications increasingly rely on complex deep learning models and large datasets. There has been an exponential growth in the amount of computation needed to train the largest models. Therefore, to scale computation and data, these models are inevitably trained in a distributed manner in clusters of nodes, and their updates are aggregated before being applied to the model. However, a distributed setup is prone to Byzantine failures of individual nodes, components, and software. With data augmentation added to these settings, there is a critical need for robust and efficient aggregation systems. We define the quality of workers as reconstruction ratios $\in (0,1]$, and formulate aggregation as a Maximum Likelihood Estimation procedure using Beta densities. We show that the Regularized form of log-likelihood wrt subspace can be approximately solved using iterative least squares solver, and provide convergence guarantees using recent Convex Optimization landscape results. Our empirical findings demonstrate that our approach significantly enhances the robustness of state-of-the-art Byzantine resilient aggregators. We evaluate our method in a distributed setup with a parameter server, and show simultaneous improvements in communication efficiency and accuracy across various tasks. The code is publicly available at https://github.com/hamidralmasi/FlagAggregator