FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models
This work addresses the problem of resource-intensive federated learning for large models, enabling efficient domain adaptation in decentralized settings, though it is incremental as it builds on existing FL and distillation techniques.
FedPromo tackles the computational burden of adapting large foundation models in federated learning by optimizing lightweight proxy models at the edge, reducing overhead while maintaining privacy, and it outperforms existing methods on five image classification benchmarks with limited-resource clients.
Federated Learning (FL) is an established paradigm for training deep learning models on decentralized data. However, as the size of the models grows, conventional FL approaches often require significant computational resources on client devices, which may not be feasible. We introduce FedPromo, a novel framework that enables efficient adaptation of large-scale foundation models stored on a central server to new domains encountered only by remote clients. Instead of directly training the large model on client devices, FedPromo optimizes lightweight proxy models via FL, significantly reducing computational overhead while maintaining privacy. Our method follows a two-stage process: first, server-side knowledge distillation aligns the representations of a large-scale foundation model (e.g., a transformer) with those of a compact counterpart (e.g., a CNN). Then, the compact model encoder is deployed to client devices, where trainable classifiers are learned locally. These classifiers are subsequently aggregated and seamlessly transferred back to the foundation model, facilitating personalized adaptation without requiring direct access to user data. Through novel regularization strategies, our framework enables decentralized multi-domain learning, balancing performance, privacy, and resource efficiency. Extensive experiments on five image classification benchmarks demonstrate that FedPromo outperforms existing methods while assuming limited-resource clients.