Training Data Protection with Compositional Diffusion Models
This addresses data privacy and customization issues for users of large-scale generative models, though it is incremental in its approach to model composition.
The paper tackles the problem of training data protection in diffusion models by introducing Compartmentalized Diffusion Models (CDM), which train separate models on distinct data sources and compose them at inference, achieving performance within 10% of a monolithic model on fine-grained vision datasets and improving text-to-image alignment by 14.33%.
We introduce Compartmentalized Diffusion Models (CDM), a method to train different diffusion models (or prompts) on distinct data sources and arbitrarily compose them at inference time. The individual models can be trained in isolation, at different times, and on different distributions and domains and can be later composed to achieve performance comparable to a paragon model trained on all data simultaneously. Furthermore, each model only contains information about the subset of the data it was exposed to during training, enabling several forms of training data protection. In particular, CDMs enable perfect selective forgetting and continual learning for large-scale diffusion models, allow serving customized models based on the user's access rights. Empirically the quality (FID) of the class-conditional CDMs (8-splits) is within 10% (on fine-grained vision datasets) of a monolithic model (no splits), and allows (8x) faster forgetting compared monolithic model with a maximum FID increase of 1%. When applied to text-to-image generation, CDMs improve alignment (TIFA) by 14.33% over a monolithic model trained on MSCOCO. CDMs also allow determining the importance of a subset of the data (attribution) in generating particular samples, and reduce memorization.