ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extraction
This addresses a practical challenge in personalized image generation for users needing to learn concepts from limited data, though it appears incremental as it builds on existing diffusion model capabilities.
The paper tackles the problem of extracting multiple concepts from a single image without human annotations, introducing the Unsupervised Concept Extraction (UCE) task and proposing ConceptExpress, which leverages pretrained diffusion models to achieve this, with experiments showing it is a promising solution.
While personalized text-to-image generation has enabled the learning of a single concept from multiple images, a more practical yet challenging scenario involves learning multiple concepts within a single image. However, existing works tackling this scenario heavily rely on extensive human annotations. In this paper, we introduce a novel task named Unsupervised Concept Extraction (UCE) that considers an unsupervised setting without any human knowledge of the concepts. Given an image that contains multiple concepts, the task aims to extract and recreate individual concepts solely relying on the existing knowledge from pretrained diffusion models. To achieve this, we present ConceptExpress that tackles UCE by unleashing the inherent capabilities of pretrained diffusion models in two aspects. Specifically, a concept localization approach automatically locates and disentangles salient concepts by leveraging spatial correspondence from diffusion self-attention; and based on the lookup association between a concept and a conceptual token, a concept-wise optimization process learns discriminative tokens that represent each individual concept. Finally, we establish an evaluation protocol tailored for the UCE task. Extensive experiments demonstrate that ConceptExpress is a promising solution to the UCE task. Our code and data are available at: https://github.com/haoosz/ConceptExpress