Q-SHED: Distributed Optimization at the Edge via Hessian Eigenvectors Quantization
This work addresses communication bottlenecks in distributed optimization for edge devices, offering incremental improvements in efficiency for scenarios like federated learning.
The paper tackles the problem of communication-efficient distributed optimization in edge networks by proposing Q-SHED, a Newton-type algorithm with Hessian eigenvectors quantization, which reduces communication rounds by up to 60% compared to competing approaches.
Edge networks call for communication efficient (low overhead) and robust distributed optimization (DO) algorithms. These are, in fact, desirable qualities for DO frameworks, such as federated edge learning techniques, in the presence of data and system heterogeneity, and in scenarios where internode communication is the main bottleneck. Although computationally demanding, Newton-type (NT) methods have been recently advocated as enablers of robust convergence rates in challenging DO problems where edge devices have sufficient computational power. Along these lines, in this work we propose Q-SHED, an original NT algorithm for DO featuring a novel bit-allocation scheme based on incremental Hessian eigenvectors quantization. The proposed technique is integrated with the recent SHED algorithm, from which it inherits appealing features like the small number of required Hessian computations, while being bandwidth-versatile at a bit-resolution level. Our empirical evaluation against competing approaches shows that Q-SHED can reduce by up to 60% the number of communication rounds required for convergence.