Testing Relational Understanding in Text-Guided Image Generation
This work highlights a critical limitation in text-guided image generation models for AI systems aiming for human-like perception and reasoning, though it is incremental as it builds on existing literature.
The researchers tested DALL-E 2's ability to generate images from prompts involving 15 basic physical and social relations, finding that only about 22% of images matched the prompts, indicating a lack of relational understanding.
Relations are basic building blocks of human cognition. Classic and recent work suggests that many relations are early developing, and quickly perceived. Machine models that aspire to human-level perception and reasoning should reflect the ability to recognize and reason generatively about relations. We report a systematic empirical examination of a recent text-guided image generation model (DALL-E 2), using a set of 15 basic physical and social relations studied or proposed in the literature, and judgements from human participants (N = 169). Overall, we find that only ~22% of images matched basic relation prompts. Based on a quantitative examination of people's judgments, we suggest that current image generation models do not yet have a grasp of even basic relations involving simple objects and agents. We examine reasons for model successes and failures, and suggest possible improvements based on computations observed in biological intelligence.