Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning
This addresses the problem of costly annotation for vision-language model tuning, offering a more efficient approach for researchers and practitioners, though it is incremental as it builds on existing pre-training methods.
The paper tackles the high labeling cost of improving zero-shot reasoning in generative vision-language models by introducing Image-Conditioned Caption Correction (ICCC), a pre-training task that enhances zero-shot performance without labeled task-aware data, showing significant improvements on models like BLIP-2 and InstructBLIP.
Generative vision-language models (VLMs) have shown impressive performance in zero-shot vision-language tasks like image captioning and visual question answering. However, improving their zero-shot reasoning typically requires second-stage instruction tuning, which relies heavily on human-labeled or large language model-generated annotation, incurring high labeling costs. To tackle this challenge, we introduce Image-Conditioned Caption Correction (ICCC), a novel pre-training task designed to enhance VLMs' zero-shot performance without the need for labeled task-aware data. The ICCC task compels VLMs to rectify mismatches between visual and language concepts, thereby enhancing instruction following and text generation conditioned on visual inputs. Leveraging language structure and a lightweight dependency parser, we construct data samples of ICCC task from image-text datasets with low labeling and computation costs. Experimental results on BLIP-2 and InstructBLIP demonstrate significant improvements in zero-shot image-text generation-based VL tasks through ICCC instruction tuning.