Synthesizing Visual Concepts as Vision-Language Programs
This addresses inconsistent reasoning in vision-language models for AI applications, offering an incremental improvement by integrating existing methods.
The paper tackles the problem of vision-language models failing at systematic visual reasoning tasks by proposing Vision-Language Programs (VLP), which combine perceptual flexibility with program synthesis to achieve consistent and interpretable outputs, outperforming direct and structured prompting on complex logical reasoning tasks.
Vision-Language models (VLMs) achieve strong performance on multimodal tasks but often fail at systematic visual reasoning tasks, leading to inconsistent or illogical outputs. Neuro-symbolic methods promise to address this by inducing interpretable logical rules, though they exploit rigid, domain-specific perception modules. We propose Vision-Language Programs (VLP), which combine the perceptual flexibility of VLMs with systematic reasoning of program synthesis. Rather than embedding reasoning inside the VLM, VLP leverages the model to produce structured visual descriptions that are compiled into neuro-symbolic programs. The resulting programs execute directly on images, remain consistent with task constraints, and provide human-interpretable explanations that enable easy shortcut mitigation. Experiments on synthetic and real-world datasets demonstrate that VLPs outperform direct and structured prompting, particularly on tasks requiring complex logical reasoning.