Trustworthiness of Stochastic Gradient Descent in Distributed Learning
This addresses privacy concerns for users of distributed learning systems, though it appears incremental as it focuses on evaluating existing compressed SGD techniques.
The paper tackled the trustworthiness of compressed Stochastic Gradient Descent in distributed learning by evaluating its resistance to privacy attacks, finding that compressed SGD shows significantly higher resistance to gradient inversion attacks compared to uncompressed SGD.
Distributed learning (DL) uses multiple nodes to accelerate training, enabling efficient optimization of large-scale models. Stochastic Gradient Descent (SGD), a key optimization algorithm, plays a central role in this process. However, communication bottlenecks often limit scalability and efficiency, leading to increasing adoption of compressed SGD techniques to alleviate these challenges. Despite addressing communication overheads, compressed SGD introduces trustworthiness concerns, as gradient exchanges among nodes are vulnerable to attacks like gradient inversion (GradInv) and membership inference attacks (MIA). The trustworthiness of compressed SGD remains unexplored, leaving important questions about its reliability unanswered. In this paper, we provide a trustworthiness evaluation of compressed versus uncompressed SGD. Specifically, we conducted empirical studies using GradInv attacks, revealing that compressed SGD demonstrates significantly higher resistance to privacy leakage compared to uncompressed SGD. In addition, our findings suggest that MIA may not be a reliable metric for assessing privacy risks in distributed learning.