Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning
This work highlights a critical limitation in current VLM training paradigms, demonstrating that scaling alone does not address reasoning deficiencies, which is significant for researchers and practitioners aiming to develop more robust VLMs.
This paper investigates the impact of reporting bias in training data on Vision-Language Models (VLMs), finding that web-scale and synthetically generated corpora insufficiently represent spatial, temporal, negation, and counting reasoning skills. The authors demonstrate that VLMs perform poorly on these skills, and scaling data, model size, or languages does not improve them, though incorporating specifically collected tacit information is effective.
The lack of reasoning capabilities in Vision-Language Models (VLMs) has remained at the forefront of research discourse. We posit that this behavior stems from a reporting bias in their training data. That is, how people communicate about visual content by default omits tacit information needed to supervise some types of reasoning; e.g., "at the game today!" is a more likely caption than "a photo of 37 people standing behind a field". We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics, and find that reporting bias results in insufficient representation of four reasoning skills (spatial, temporal, negation, and counting), despite the corpora being of web-scale, and/or synthetically generated. With a set of curated benchmarks, we demonstrate that: (i) VLMs perform poorly on the aforementioned types of reasoning suppressed in the training data by reporting bias; (ii) contrary to popular belief, scaling data size, model size, and to multiple languages does not result in emergence of these skills by default; but, promisingly, (iii) incorporating annotations specifically collected to obtain tacit information is effective. Our findings highlight the need for more intentional training data curation methods, rather than counting on scale for emergence of reasoning capabilities.