Per-Query Visual Concept Learning
This work addresses the need for more effective text-to-image personalization, which is incremental as it builds upon and improves six prior methods.
The paper tackles the problem of visual concept learning by introducing a per-query personalization step that uses attention-based loss terms to improve identity capture, resulting in significant enhancements over existing methods across multiple models.
Visual concept learning, also known as Text-to-image personalization, is the process of teaching new concepts to a pretrained model. This has numerous applications from product placement to entertainment and personalized design. Here we show that many existing methods can be substantially augmented by adding a personalization step that is (1) specific to the prompt and noise seed, and (2) using two loss terms based on the self- and cross- attention, capturing the identity of the personalized concept. Specifically, we leverage PDM features -- previously designed to capture identity -- and show how they can be used to improve personalized semantic similarity. We evaluate the benefit that our method gains on top of six different personalization methods, and several base text-to-image models (both UNet- and DiT-based). We find significant improvements even over previous per-query personalization methods.