PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis
This work addresses the problem of inconsistent semantics and layout in image synthesis for applications like content creation, but it is incremental as it builds on existing pre-trained models with novel fusion and loss techniques.
The paper tackles the challenge of synthesizing high-quality images with consistent semantics and layout in semantic image synthesis by proposing PLACE, an adaptive layout-semantic fusion module that uses layout control maps and timestep-adaptive feature fusion, along with Semantic Alignment and Layout-Free Prior Preservation losses during fine-tuning, resulting in improved visual quality, semantic consistency, and layout alignment as demonstrated in experiments.
Recent advancements in large-scale pre-trained text-to-image models have led to remarkable progress in semantic image synthesis. Nevertheless, synthesizing high-quality images with consistent semantics and layout remains a challenge. In this paper, we propose the adaPtive LAyout-semantiC fusion modulE (PLACE) that harnesses pre-trained models to alleviate the aforementioned issues. Specifically, we first employ the layout control map to faithfully represent layouts in the feature space. Subsequently, we combine the layout and semantic features in a timestep-adaptive manner to synthesize images with realistic details. During fine-tuning, we propose the Semantic Alignment (SA) loss to further enhance layout alignment. Additionally, we introduce the Layout-Free Prior Preservation (LFP) loss, which leverages unlabeled data to maintain the priors of pre-trained models, thereby improving the visual quality and semantic consistency of synthesized images. Extensive experiments demonstrate that our approach performs favorably in terms of visual quality, semantic consistency, and layout alignment. The source code and model are available at https://github.com/cszy98/PLACE/tree/main.