FedMAE: Federated Self-Supervised Learning with One-Block Masked Auto-Encoder
This addresses the problem of scalable self-supervised learning in federated settings for applications like image analysis, though it is incremental as it builds on existing masked autoencoder and federated learning techniques.
The paper tackles the challenge of training on unlabeled large-scale images in federated learning with limited client resources by introducing FedMAE, a framework that pre-trains one-block masked autoencoders on clients and cascades them on a server, achieving superior performance in image reconstruction and classification compared to state-of-the-art methods.
Latest federated learning (FL) methods started to focus on how to use unlabeled data in clients for training due to users' privacy concerns, high labeling costs, or lack of expertise. However, current Federated Semi-Supervised/Self-Supervised Learning (FSSL) approaches fail to learn large-scale images because of the limited computing resources of local clients. In this paper, we introduce a new framework FedMAE, which stands for Federated Masked AutoEncoder, to address the problem of how to utilize unlabeled large-scale images for FL. Specifically, FedMAE can pre-train one-block Masked AutoEncoder (MAE) using large images in lightweight client devices, and then cascades multiple pre-trained one-block MAEs in the server to build a multi-block ViT backbone for downstream tasks. Theoretical analysis and experimental results on image reconstruction and classification show that our FedMAE achieves superior performance compared to the state-of-the-art FSSL methods.