SpotDiff: Spotting and Disentangling Interference in Feature Space for Subject-Preserving Image Generation
This addresses the challenge of subject identity preservation in image generation for users needing efficient and high-fidelity results, representing an incremental improvement over existing methods.
The paper tackles the problem of preserving a subject's identity in personalized image generation by introducing SpotDiff, a learning-based method that spots and disentangles interference in feature space, achieving robust subject preservation and controllable editing with competitive performance using only 10k training samples.
Personalized image generation aims to faithfully preserve a reference subject's identity while adapting to diverse text prompts. Existing optimization-based methods ensure high fidelity but are computationally expensive, while learning-based approaches offer efficiency at the cost of entangled representations influenced by nuisance factors. We introduce SpotDiff, a novel learning-based method that extracts subject-specific features by spotting and disentangling interference. Leveraging a pre-trained CLIP image encoder and specialized expert networks for pose and background, SpotDiff isolates subject identity through orthogonality constraints in the feature space. To enable principled training, we introduce SpotDiff10k, a curated dataset with consistent pose and background variations. Experiments demonstrate that SpotDiff achieves more robust subject preservation and controllable editing than prior methods, while attaining competitive performance with only 10k training samples.