Secure and Fast Asynchronous Vertical Federated Learning via Cascaded Hybrid Optimization
This work addresses a bottleneck in privacy-preserving VFL for large models, making it more practical for applications with vertically partitioned data, though it is incremental as it builds on existing ZOO and FOO techniques.
The paper tackles the slow convergence of zeroth-order optimization (ZOO) in vertical federated learning (VFL) by proposing a cascaded hybrid method that uses ZOO for clients to protect privacy and first-order optimization (FOO) for the server to speed up training, achieving faster convergence comparable to unsafe FOO-based baselines while maintaining privacy.
Vertical Federated Learning (VFL) attracts increasing attention because it empowers multiple parties to jointly train a privacy-preserving model over vertically partitioned data. Recent research has shown that applying zeroth-order optimization (ZOO) has many advantages in building a practical VFL algorithm. However, a vital problem with the ZOO-based VFL is its slow convergence rate, which limits its application in handling modern large models. To address this problem, we propose a cascaded hybrid optimization method in VFL. In this method, the downstream models (clients) are trained with ZOO to protect privacy and ensure that no internal information is shared. Meanwhile, the upstream model (server) is updated with first-order optimization (FOO) locally, which significantly improves the convergence rate, making it feasible to train the large models without compromising privacy and security. We theoretically prove that our VFL framework converges faster than the ZOO-based VFL, as the convergence of our framework is not limited by the size of the server model, making it effective for training large models with the major part on the server. Extensive experiments demonstrate that our method achieves faster convergence than the ZOO-based VFL framework, while maintaining an equivalent level of privacy protection. Moreover, we show that the convergence of our VFL is comparable to the unsafe FOO-based VFL baseline. Additionally, we demonstrate that our method makes the training of a large model feasible.