Variational Federated Multi-Task Learning
This work addresses statistical heterogeneity in federated learning for non-convex models, which is incremental as it extends existing multi-task learning to a broader class of models.
The paper tackled the problem of federated multi-task learning for non-convex models, which had only been applied to convex models before, and introduced VIRTUAL, a variational inference-based algorithm that outperforms state-of-the-art federated learning methods on real-world datasets while enabling sparser gradient updates.
In federated learning, a central server coordinates the training of a single model on a massively distributed network of devices. This setting can be naturally extended to a multi-task learning framework, to handle real-world federated datasets that typically show strong statistical heterogeneity among devices. Despite federated multi-task learning being shown to be an effective paradigm for real-world datasets, it has been applied only on convex models. In this work, we introduce VIRTUAL, an algorithm for federated multi-task learning for general non-convex models. In VIRTUAL the federated network of the server and the clients is treated as a star-shaped Bayesian network, and learning is performed on the network using approximated variational inference. We show that this method is effective on real-world federated datasets, outperforming the current state-of-the-art for federated learning, and concurrently allowing sparser gradient updates.