Label-free Neural Semantic Image Synthesis
This addresses the need for fine-grained spatial control in image synthesis without costly manual annotations, offering a label-free alternative that can augment real data for perception tasks.
The paper tackles the problem of expensive or ambiguous spatial conditioning in text-to-image diffusion models by proposing neural semantic image synthesis, which uses neural layouts from pre-trained foundation models as conditioning, achieving similar or superior pixel-level alignment compared to semantic label maps while capturing better semantics, instance separation, and object orientation than other label-free options.
Recent work has shown great progress in integrating spatial conditioning to control large, pre-trained text-to-image diffusion models. Despite these advances, existing methods describe the spatial image content using hand-crafted conditioning inputs, which are either semantically ambiguous (e.g., edges) or require expensive manual annotations (e.g., semantic segmentation). To address these limitations, we propose a new label-free way of conditioning diffusion models to enable fine-grained spatial control. We introduce the concept of neural semantic image synthesis, which uses neural layouts extracted from pre-trained foundation models as conditioning. Neural layouts are advantageous as they provide rich descriptions of the desired image, containing both semantics and detailed geometry of the scene. We experimentally show that images synthesized via neural semantic image synthesis achieve similar or superior pixel-level alignment of semantic classes compared to those created using expensive semantic label maps. At the same time, they capture better semantics, instance separation, and object orientation than other label-free conditioning options, such as edges or depth. Moreover, we show that images generated by neural layout conditioning can effectively augment real data for training various perception tasks.