Semantic-shape Adaptive Feature Modulation for Semantic Image Synthesis
This work addresses the problem of enhancing detail generation in semantic image synthesis for applications like computer vision and graphics, representing an incremental improvement by building on existing methods with novel feature modulation.
The paper tackles the challenge of generating photo-realistic images with rich details in semantic image synthesis by introducing a Shape-aware Position Descriptor (SPD) and a Semantic-shape Adaptive Feature Modulation (SAFM) block to exploit part-level layouts inferred from object shapes, resulting in significant improvements in object detail generation and favorable performance against state-of-the-art methods.
Recent years have witnessed substantial progress in semantic image synthesis, it is still challenging in synthesizing photo-realistic images with rich details. Most previous methods focus on exploiting the given semantic map, which just captures an object-level layout for an image. Obviously, a fine-grained part-level semantic layout will benefit object details generation, and it can be roughly inferred from an object's shape. In order to exploit the part-level layouts, we propose a Shape-aware Position Descriptor (SPD) to describe each pixel's positional feature, where object shape is explicitly encoded into the SPD feature. Furthermore, a Semantic-shape Adaptive Feature Modulation (SAFM) block is proposed to combine the given semantic map and our positional features to produce adaptively modulated features. Extensive experiments demonstrate that the proposed SPD and SAFM significantly improve the generation of objects with rich details. Moreover, our method performs favorably against the SOTA methods in terms of quantitative and qualitative evaluation. The source code and model are available at https://github.com/cszy98/SAFM.