FineControlNet: Fine-level Text Control for Image Generation with Spatially Aligned Text Control Injection
This work addresses the limitation of existing models in providing fine-grained appearance control for image generation, which is incremental but useful for applications requiring detailed scene composition.
The paper tackles the problem of controlling both the geometric pose and visual appearance of individual instances in text-to-image generation, achieving superior performance in generating images that follow user-provided instance-specific text prompts and poses compared to existing methods.
Recently introduced ControlNet has the ability to steer the text-driven image generation process with geometric input such as human 2D pose, or edge features. While ControlNet provides control over the geometric form of the instances in the generated image, it lacks the capability to dictate the visual appearance of each instance. We present FineControlNet to provide fine control over each instance's appearance while maintaining the precise pose control capability. Specifically, we develop and demonstrate FineControlNet with geometric control via human pose images and appearance control via instance-level text prompts. The spatial alignment of instance-specific text prompts and 2D poses in latent space enables the fine control capabilities of FineControlNet. We evaluate the performance of FineControlNet with rigorous comparison against state-of-the-art pose-conditioned text-to-image diffusion models. FineControlNet achieves superior performance in generating images that follow the user-provided instance-specific text prompts and poses compared with existing methods. Project webpage: https://samsunglabs.github.io/FineControlNet-project-page