The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models
This work addresses the need for better evaluation of multimodal models' basic language abilities, which is crucial for researchers and developers in AI and NLP, though it is incremental as it builds on existing benchmarks and findings.
The authors tackled the problem of evaluating whether pre-trained multimodal models truly understand basic linguistic constructions like active-passive voice, coordination, and relative clauses, by introducing the BLA benchmark, and found that models such as CLIP, ViLBERT, and BLIP2 generally struggle with it, with BLIP2 showing some improvement in in-context learning.
Despite the impressive performance achieved by pre-trained language-and-vision models in downstream tasks, it remains an open question whether this reflects a proper understanding of image-text interaction. In this work, we explore to what extent they handle basic linguistic constructions -- active-passive voice, coordination, and relative clauses -- that even preschool children can typically master. We present BLA, a novel, automatically constructed benchmark to evaluate multimodal models on these Basic Language Abilities. We show that different types of Transformer-based systems, such as CLIP, ViLBERT, and BLIP2, generally struggle with BLA in a zero-shot setting, in line with previous findings. Our experiments, in particular, show that most of the tested models only marginally benefit when fine-tuned or prompted with construction-specific samples. Yet, the generative BLIP2 shows promising trends, especially in an in-context learning setting. This opens the door to using BLA not only as an evaluation benchmark but also to improve models' basic language abilities.