What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge
This work addresses the need for benchmarks to evaluate visual commonsense learning in AI models, but it is incremental as it shows limited impact from current multimodal approaches.
The paper tackled the problem of measuring whether multimodal training improves visual commonsense knowledge in language models, and found no significant difference between multimodal models and unimodal baselines trained on visual text data.
There are limitations in learning language from text alone. Therefore, recent focus has been on developing multimodal models. However, few benchmarks exist that can measure what language models learn about language from multimodal training. We hypothesize that training on a visual modality should improve on the visual commonsense knowledge in language models. Therefore, we introduce two evaluation tasks for measuring visual commonsense knowledge in language models and use them to evaluate different multimodal models and unimodal baselines. Primarily, we find that the visual commonsense knowledge is not significantly different between the multimodal models and unimodal baseline models trained on visual text data.