How to Determine the Preferred Image Distribution of a Black-Box Vision-Language Model?
This addresses the challenge of improving VLM performance in complex visual reasoning tasks for specialized fields such as CAD, though it appears incremental by refining existing methods and introducing a new benchmark.
The paper tackles the problem of optimizing black-box Vision-Language Models (VLMs) for specialized visual tasks by proposing a method to identify preferred image distributions, demonstrating its efficacy in domains like Computer-Aided Design (CAD) and introducing the CAD-VQA dataset for evaluation.
Large foundation models have revolutionized the field, yet challenges remain in optimizing multi-modal models for specialized visual tasks. We propose a novel, generalizable methodology to identify preferred image distributions for black-box Vision-Language Models (VLMs) by measuring output consistency across varied input prompts. Applying this to different rendering types of 3D objects, we demonstrate its efficacy across various domains requiring precise interpretation of complex structures, with a focus on Computer-Aided Design (CAD) as an exemplar field. We further refine VLM outputs using in-context learning with human feedback, significantly enhancing explanation quality. To address the lack of benchmarks in specialized domains, we introduce CAD-VQA, a new dataset for evaluating VLMs on CAD-related visual question answering tasks. Our evaluation of state-of-the-art VLMs on CAD-VQA establishes baseline performance levels, providing a framework for advancing VLM capabilities in complex visual reasoning tasks across various fields requiring expert-level visual interpretation. We release the dataset and evaluation codes at \url{https://github.com/asgsaeid/cad_vqa}.