What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation
This work addresses the challenge of controllable image generation for applications requiring accurate spatial relationships and object interactions, representing a domain-specific advancement.
The authors tackled the problem of evaluating and improving factual consistency in scene graph-to-image generation by introducing Scene-Bench, a benchmark with a large-scale dataset and a novel evaluation metric called SGScore. Their feedback pipeline significantly enhanced factual consistency in generated images, though specific numerical improvements were not provided in the abstract.
While text-to-image generation has been extensively studied, generating images from scene graphs remains relatively underexplored, primarily due to challenges in accurately modeling spatial relationships and object interactions. To fill this gap, we introduce Scene-Bench, a comprehensive benchmark designed to evaluate and enhance the factual consistency in generating natural scenes. Scene-Bench comprises MegaSG, a large-scale dataset of one million images annotated with scene graphs, facilitating the training and fair comparison of models across diverse and complex scenes. Additionally, we propose SGScore, a novel evaluation metric that leverages chain-of-thought reasoning capabilities of multimodal large language models (LLMs) to assess both object presence and relationship accuracy, offering a more effective measure of factual consistency than traditional metrics like FID and CLIPScore. Building upon this evaluation framework, we develop a scene graph feedback pipeline that iteratively refines generated images by identifying and correcting discrepancies between the scene graph and the image. Extensive experiments demonstrate that Scene-Bench provides a more comprehensive and effective evaluation framework compared to existing benchmarks, particularly for complex scene generation. Furthermore, our feedback strategy significantly enhances the factual consistency of image generation models, advancing the field of controllable image generation.