Diffusion-based image inpainting with internal learning
This addresses the problem of computational inefficiency in diffusion models for image inpainting, particularly for domains with unique image acquisition modalities, though it is incremental as it builds on existing diffusion methods.
The paper tackles the high computational cost of diffusion models for image inpainting by proposing lightweight models that can be trained on a single or few images, achieving competitive results with large state-of-the-art models in specific cases like texture images, line drawings, and materials BRDF, with greatly reduced computational load.
Diffusion models are now the undisputed state-of-the-art for image generation and image restoration. However, they require large amounts of computational power for training and inference. In this paper, we propose lightweight diffusion models for image inpainting that can be trained on a single image, or a few images. We show that our approach competes with large state-of-the-art models in specific cases. We also show that training a model on a single image is particularly relevant for image acquisition modality that differ from the RGB images of standard learning databases. We show results in three different contexts: texture images, line drawing images, and materials BRDF, for which we achieve state-of-the-art results in terms of realism, with a computational load that is greatly reduced compared to concurrent methods.