FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learning
This addresses privacy-preserving adaptation of T2I models for specialized applications, but appears incremental as it builds on federated learning concepts.
The paper tackles the problem of adapting text-to-image models to specialized tasks with limited task-specific data due to privacy concerns, by introducing FedMMKT, a framework that co-enhances a server T2I model and client task models using decentralized multimodal data without compromising privacy.
Text-to-Image (T2I) models have demonstrated their versatility in a wide range of applications. However, adaptation of T2I models to specialized tasks is often limited by the availability of task-specific data due to privacy concerns. On the other hand, harnessing the power of rich multimodal data from modern mobile systems and IoT infrastructures presents a great opportunity. This paper introduces Federated Multi-modal Knowledge Transfer (FedMMKT), a novel framework that enables co-enhancement of a server T2I model and client task-specific models using decentralized multimodal data without compromising data privacy.