Diffusion-based Holistic Texture Rectification and Synthesis
This addresses a domain-specific problem in computer vision for texture synthesis, enabling more practical applications from natural images, but it appears incremental as it builds on existing diffusion models and texture synthesis techniques.
The paper tackles the problem of synthesizing holistic textures from degraded samples in natural images, which often have occlusions and distortions, by proposing a framework that uses a conditional Latent Diffusion Model with an occlusion-aware latent transformer. Experimental results show it significantly outperforms existing methods both quantitatively and qualitatively.
We present a novel framework for rectifying occlusions and distortions in degraded texture samples from natural images. Traditional texture synthesis approaches focus on generating textures from pristine samples, which necessitate meticulous preparation by humans and are often unattainable in most natural images. These challenges stem from the frequent occlusions and distortions of texture samples in natural images due to obstructions and variations in object surface geometry. To address these issues, we propose a framework that synthesizes holistic textures from degraded samples in natural images, extending the applicability of exemplar-based texture synthesis techniques. Our framework utilizes a conditional Latent Diffusion Model (LDM) with a novel occlusion-aware latent transformer. This latent transformer not only effectively encodes texture features from partially-observed samples necessary for the generation process of the LDM, but also explicitly captures long-range dependencies in samples with large occlusions. To train our model, we introduce a method for generating synthetic data by applying geometric transformations and free-form mask generation to clean textures. Experimental results demonstrate that our framework significantly outperforms existing methods both quantitatively and quantitatively. Furthermore, we conduct comprehensive ablation studies to validate the different components of our proposed framework. Results are corroborated by a perceptual user study which highlights the efficiency of our proposed approach.