PhyCustom: Towards Realistic Physical Customization in Text-to-Image Generation
This addresses a realistic yet challenging customization problem for users of text-to-image models, representing an incremental improvement by introducing physical knowledge into existing diffusion models.
The paper tackles the problem of accurately reflecting physical properties in text-to-image generation, where existing methods fail despite physics-related prompts, and proposes PhyCustom, a fine-tuning framework that outperforms state-of-the-art methods in physical customization quantitatively and qualitatively.
Recent diffusion-based text-to-image customization methods have achieved significant success in understanding concrete concepts to control generation processes, such as styles and shapes. However, few efforts dive into the realistic yet challenging customization of physical concepts. The core limitation of current methods arises from the absence of explicitly introducing physical knowledge during training. Even when physics-related words appear in the input text prompts, our experiments consistently demonstrate that these methods fail to accurately reflect the corresponding physical properties in the generated results. In this paper, we propose PhyCustom, a fine-tuning framework comprising two novel regularization losses to activate diffusion model to perform physical customization. Specifically, the proposed isometric loss aims at activating diffusion models to learn physical concepts while decouple loss helps to eliminate the mixture learning of independent concepts. Experiments are conducted on a diverse dataset and our benchmark results demonstrate that PhyCustom outperforms previous state-of-the-art and popular methods in terms of physical customization quantitatively and qualitatively.