Desirable Companion for Vertical Federated Learning: New Zeroth-Order Gradient Based Algorithm
This work addresses the problem of privacy leakage and inefficiency in multi-party collaborative modeling for vertical federated learning, offering a practical solution that is incremental in integrating ZOO into VFL.
The paper tackles the challenge of designing a vertical federated learning (VFL) algorithm that meets all key criteria—model applicability, privacy security, communication cost, and computation efficiency—by proposing a novel VFL framework based on zeroth-order optimization (ZOO), which improves model applicability, prevents privacy leakage under various threat models, and supports inexpensive communication and efficient computation, with extensive experiments on benchmark datasets demonstrating favorable performance across these metrics.
Vertical federated learning (VFL) attracts increasing attention due to the emerging demands of multi-party collaborative modeling and concerns of privacy leakage. A complete list of metrics to evaluate VFL algorithms should include model applicability, privacy security, communication cost, and computation efficiency, where privacy security is especially important to VFL. However, to the best of our knowledge, there does not exist a VFL algorithm satisfying all these criteria very well. To address this challenging problem, in this paper, we reveal that zeroth-order optimization (ZOO) is a desirable companion for VFL. Specifically, ZOO can 1) improve the model applicability of VFL framework, 2) prevent VFL framework from privacy leakage under curious, colluding, and malicious threat models, 3) support inexpensive communication and efficient computation. Based on that, we propose a novel and practical VFL framework with black-box models, which is inseparably interconnected to the promising properties of ZOO. We believe that it takes one stride towards designing a practical VFL framework matching all the criteria. Under this framework, we raise two novel {\bf asy}nchronous ze{\bf r}oth-ord{\bf e}r algorithms for {\bf v}ertical f{\bf e}derated {\bf l}earning (AsyREVEL) with different smoothing techniques. We theoretically drive the convergence rates of AsyREVEL algorithms under nonconvex condition. More importantly, we prove the privacy security of our proposed framework under existing VFL attacks on different levels. Extensive experiments on benchmark datasets demonstrate the favorable model applicability, satisfied privacy security, inexpensive communication, efficient computation, scalability and losslessness of our framework.