ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
This work addresses efficiency and robustness issues in multi-party learning for scenarios with scattered data under legal constraints, representing an incremental improvement through a novel sampling strategy.
The paper tackles the challenge of skewed data distributions and computational bottlenecks in distributed multi-party learning by proposing ADDS, an adaptive differentiable sampling framework that reduces local computation and communication costs while speeding up central model convergence, as demonstrated on real-world datasets.
Distributed multi-party learning provides an effective approach for training a joint model with scattered data under legal and practical constraints. However, due to the quagmire of a skewed distribution of data labels across participants and the computation bottleneck of local devices, how to build smaller customized models for clients in various scenarios while providing updates appliable to the central model remains a challenge. In this paper, we propose a novel adaptive differentiable sampling framework (ADDS) for robust and communication-efficient multi-party learning. Inspired by the idea of dropout in neural networks, we introduce a network sampling strategy in the multi-party setting, which distributes different subnets of the central model to clients for updating, and the differentiable sampling rates allow each client to extract optimal local architecture from the supernet according to its private data distribution. The approach requires minimal modifications to the existing multi-party learning structure, and it is capable of integrating local updates of all subnets into the supernet, improving the robustness of the central model. The proposed framework significantly reduces local computation and communication costs while speeding up the central model convergence, as we demonstrated through experiments on real-world datasets.