What Decreases Editing Capability? Domain-Specific Hybrid Refinement for Improved GAN Inversion
This addresses a specific bottleneck in GAN inversion for image editing, particularly for complex images, but is incremental as it refines existing techniques.
The paper tackles the problem of GAN inversion methods decreasing editing capability while improving reconstruction, especially on complex images, and introduces Domain-Specific Hybrid Refinement (DHR) to maintain editability with fidelity gains, achieving state-of-the-art results in real image inversion and editing.
Recently, inversion methods have focused on additional high-rate information in the generator (e.g., weights or intermediate features) to refine inversion and editing results from embedded latent codes. Although these techniques gain reasonable improvement in reconstruction, they decrease editing capability, especially on complex images (e.g., containing occlusions, detailed backgrounds, and artifacts). A vital crux is refining inversion results, avoiding editing capability degradation. To tackle this problem, we introduce Domain-Specific Hybrid Refinement (DHR), which draws on the advantages and disadvantages of two mainstream refinement techniques to maintain editing ability with fidelity improvement. Specifically, we first propose Domain-Specific Segmentation to segment images into two parts: in-domain and out-of-domain parts. The refinement process aims to maintain the editability for in-domain areas and improve two domains' fidelity. We refine these two parts by weight modulation and feature modulation, which we call Hybrid Modulation Refinement. Our proposed method is compatible with all latent code embedding methods. Extension experiments demonstrate that our approach achieves state-of-the-art in real image inversion and editing. Code is available at https://github.com/caopulan/Domain-Specific_Hybrid_Refinement_Inversion.