A Framework for Double-Blind Federated Adaptation of Foundation Models
This addresses privacy concerns in federated learning for foundation models, enabling secure adaptation without data sharing, though it is incremental in combining existing privacy techniques with new architectural modifications.
The paper tackles the problem of adapting foundation models collaboratively without compromising privacy, by proposing BlindFed, a framework that uses fully homomorphic encryption and novel techniques to protect both data owners and the learning service provider, achieving practical feasibility on image classification datasets with high communication and computational costs.
Foundation models (FMs) excel in zero-shot tasks but benefit from task-specific adaptation. However, privacy concerns prevent data sharing among multiple data owners, and proprietary restrictions prevent the learning service provider (LSP) from sharing the FM. In this work, we propose BlindFed, a framework enabling collaborative FM adaptation while protecting both parties: data owners do not access the FM or each other's data, and the LSP does not see sensitive task data. BlindFed relies on fully homomorphic encryption (FHE) and consists of three key innovations: (i) FHE-friendly architectural modifications via polynomial approximations and low-rank adapters, (ii) a two-stage split learning approach combining offline knowledge distillation and online encrypted inference for adapter training without backpropagation through the FM, and (iii) a privacy-boosting scheme using sample permutations and stochastic block sampling to mitigate model extraction attacks. Empirical results on four image classification datasets demonstrate the practical feasibility of the BlindFed framework, albeit at a high communication cost and large computational complexity for the LSP.