PropTest: Automatic Property Testing for Improved Visual Programming
This work addresses the challenge of enhancing interpretability and performance in visual reasoning for AI applications, representing an incremental improvement over existing visual programming techniques.
The paper tackles the problem of improving visual programming methods by introducing PropTest, a strategy that uses LLMs to generate tests for visual properties in proposed solutions, achieving comparable results to state-of-the-art methods with publicly available LLMs, such as improving ViperGPT by 6.0% to 46.1% accuracy on GQA and 8.1% to 59.5% on RefCOCO+.
Visual Programming has recently emerged as an alternative to end-to-end black-box visual reasoning models. This type of method leverages Large Language Models (LLMs) to generate the source code for an executable computer program that solves a given problem. This strategy has the advantage of offering an interpretable reasoning path and does not require finetuning a model with task-specific data. We propose PropTest, a general strategy that improves visual programming by further using an LLM to generate code that tests for visual properties in an initial round of proposed solutions. Our method generates tests for data-type consistency, output syntax, and semantic properties. PropTest achieves comparable results to state-of-the-art methods while using publicly available LLMs. This is demonstrated across different benchmarks on visual question answering and referring expression comprehension. Particularly, PropTest improves ViperGPT by obtaining 46.1\% accuracy (+6.0\%) on GQA using Llama3-8B and 59.5\% (+8.1\%) on RefCOCO+ using CodeLlama-34B.