From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer Decomposition
This addresses the challenge of limited methods and data for image layer decomposition, enabling independent editing for content creation, though it is incremental as it repurposes existing inpainting models.
The paper tackles the problem of decomposing a single image into layers (foreground and background) by adapting a diffusion-based inpainting model with lightweight finetuning and a multi-modal context fusion module, achieving superior performance in object removal and occlusion recovery.
Images can be viewed as layered compositions, foreground objects over background, with potential occlusions. This layered representation enables independent editing of elements, offering greater flexibility for content creation. Despite the progress in large generative models, decomposing a single image into layers remains challenging due to limited methods and data. We observe a strong connection between layer decomposition and in/outpainting tasks, and propose adapting a diffusion-based inpainting model for layer decomposition using lightweight finetuning. To further preserve detail in the latent space, we introduce a novel multi-modal context fusion module with linear attention complexity. Our model is trained purely on a synthetic dataset constructed from open-source assets and achieves superior performance in object removal and occlusion recovery, unlocking new possibilities in downstream editing and creative applications.