DALL-E 2 Fails to Reliably Capture Common Syntactic Processes
This highlights limitations in AI language comprehension for researchers and developers, though it is incremental as it builds on existing critiques.
The study systematically tested DALL-E 2's ability to capture 8 common grammatical phenomena, such as binding and negation, and found it fails to reliably infer meanings consistent with syntax, challenging claims about its language understanding.
Machine intelligence is increasingly being linked to claims about sentience, language processing, and an ability to comprehend and transform natural language into a range of stimuli. We systematically analyze the ability of DALL-E 2 to capture 8 grammatical phenomena pertaining to compositionality that are widely discussed in linguistics and pervasive in human language: binding principles and coreference, passives, word order, coordination, comparatives, negation, ellipsis, and structural ambiguity. Whereas young children routinely master these phenomena, learning systematic mappings between syntax and semantics, DALL-E 2 is unable to reliably infer meanings that are consistent with the syntax. These results challenge recent claims concerning the capacity of such systems to understand of human language. We make available the full set of test materials as a benchmark for future testing.