Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models
This addresses the challenge of fine-tuning diffusion models on resource-constrained devices for users needing custom image generation, representing an incremental improvement over prior methods focused on training steps and parameter reduction.
The paper tackled the problem of on-device personalization of text-to-image diffusion models by proposing Hollowed Net, an efficient LoRA-based approach that reduces GPU memory requirements for training to levels as low as inference, while maintaining or improving personalization performance.
Recent advancements in text-to-image diffusion models have enabled the personalization of these models to generate custom images from textual prompts. This paper presents an efficient LoRA-based personalization approach for on-device subject-driven generation, where pre-trained diffusion models are fine-tuned with user-specific data on resource-constrained devices. Our method, termed Hollowed Net, enhances memory efficiency during fine-tuning by modifying the architecture of a diffusion U-Net to temporarily remove a fraction of its deep layers, creating a hollowed structure. This approach directly addresses on-device memory constraints and substantially reduces GPU memory requirements for training, in contrast to previous methods that primarily focus on minimizing training steps and reducing the number of parameters to update. Additionally, the personalized Hollowed Net can be transferred back into the original U-Net, enabling inference without additional memory overhead. Quantitative and qualitative analyses demonstrate that our approach not only reduces training memory to levels as low as those required for inference but also maintains or improves personalization performance compared to existing methods.