Yuhao Yi

Yuhao Yi, Zhongzhi Zhang, Stacy Patterson

The vast majority of real-world networks are scale-free, loopy, and sparse, with a power-law degree distribution and a constant average degree. In this paper, we study first-order consensus dynamics in binary scale-free networks, where vertices are subject to white noise. We focus on the coherence of networks characterized in terms of the $H_2$-norm, which quantifies how closely agents track the consensus value. We first provide a lower bound of coherence of a network in terms of its average degree, which is independent of the network order. We then study the coherence of some sparse, scale-free real-world networks, which approaches a constant. We also study numerically the coherence of Barabási-Albert networks and high-dimensional random Apollonian networks, which also converges to a constant when the networks grow. Finally, based on the connection of coherence and the Kirchhoff index, we study analytically the coherence of two deterministically-growing sparse networks and obtain the exact expressions, which tend to small constants. Our results indicate that the effect of noise on the consensus dynamics in power-law networks is negligible. We argue that scale-free topology, together with loopy structure, is responsible for the strong robustness with respect to noisy consensus dynamics in power-law networks.

Yi Qi, Zhongzhi Zhang, Yuhao Yi et al.

The hierarchical graphs and Sierpiński graphs are constructed iteratively, which have the same number of vertices and edges at any iteration, but exhibit quite different structural properties: the hierarchical graphs are non-fractal and small-world, while the Sierpiński graphs are fractal and "large-world". Both graphs have found broad applications. In this paper, we study consensus problems in hierarchical graphs and Sierpiński graphs, focusing on three important quantities of consensus problems, that is, convergence speed, delay robustness, and coherence for first-order (and second-order) dynamics, which are, respectively, determined by algebraic connectivity, maximum eigenvalue, and sum of reciprocal (and square of reciprocal) of each nonzero eigenvalue of Laplacian matrix. For both graphs, based on the explicit recursive relation of eigenvalues at two successive iterations, we evaluate the second smallest eigenvalue, as well as the largest eigenvalue, and obtain the closed-form solutions to the sum of reciprocals (and square of reciprocals) of all nonzero eigenvalues. We also compare our obtained results for consensus problems on both graphs and show that they differ in all quantities concerned, which is due to the marked difference of their topological structures.

Yuhao Yi, Ronghui You, Hong Liu et al.

Byzantine machine learning has garnered considerable attention in light of the unpredictable faults that can occur in large-scale distributed learning systems. The key to secure resilience against Byzantine machines in distributed learning is resilient aggregation mechanisms. Although abundant resilient aggregation rules have been proposed, they are designed in ad-hoc manners, imposing extra barriers on comparing, analyzing, and improving the rules across performance criteria. This paper studies near-optimal aggregation rules using clustering in the presence of outliers. Our outlier-robust clustering approach utilizes geometric properties of the update vectors provided by workers. Our analysis show that constant approximations to the 1-center and 1-mean clustering problems with outliers provide near-optimal resilient aggregators for metric-based criteria, which have been proven to be crucial in the homogeneous and heterogeneous cases respectively. In addition, we discuss two contradicting types of attacks under which no single aggregation rule is guaranteed to improve upon the naive average. Based on the discussion, we propose a two-phase resilient aggregation framework. We run experiments for image classification using a non-convex loss function. The proposed algorithms outperform previously known aggregation rules by a large margin with both homogeneous and heterogeneous data distributions among non-faulty workers. Code and appendix are available at https://github.com/jerry907/AAAI24-RASHB.

Changxin Liu, Yanghao Li, Yuhao Yi et al.

Distributed learning has become the standard approach for training large-scale machine learning models across private data silos. While distributed learning enhances privacy preservation and training efficiency, it faces critical challenges related to Byzantine robustness and communication reduction. Existing Byzantine-robust and communication-efficient methods rely on full gradient information either at every iteration or at certain iterations with a probability, and they only converge to an unnecessarily large neighborhood around the solution. Motivated by these issues, we propose a novel Byzantine-robust and communication-efficient stochastic distributed learning method that imposes no requirements on batch size and converges to a smaller neighborhood around the optimal solution than all existing methods, aligning with the theoretical lower bound. Our key innovation is leveraging Polyak Momentum to mitigate the noise caused by both biased compressors and stochastic gradients, thus defending against Byzantine workers under information compression. We provide proof of tight complexity bounds for our algorithm in the context of non-convex smooth loss functions, demonstrating that these bounds match the lower bounds in Byzantine-free scenarios. Finally, we validate the practical significance of our algorithm through an extensive series of experiments, benchmarking its performance on both binary classification and image classification tasks.

Stacy Patterson, Yuhao Yi, Zhongzhi Zhang

We study the performance of leader-follower noisy consensus networks, and in particular, the relationship between this performance and the locations of the leader nodes. Two types of dynamics are considered (1) noise-free leaders, in which leaders dictate the trajectory exactly and followers are subject to external disturbances, and (2) noise-corrupted leaders, in which both leaders and followers are subject to external perturbations. We measure the performance of a network by its coherence, an $H_2$ norm that quantifies how closely the followers track the leaders' trajectory. For both dynamics, we show a relationship between the coherence and resistance distances in an a electrical network. Using this relationship, we derive closed-form expressions for coherence as a function of the locations of the leaders. Further, we give analytical solutions to the optimal leader selection problem for several special classes of graphs.