Pseudo Contrastive Learning for Diagram Comprehension in Multimodal Models
This addresses the challenge of diagram comprehension in vision-language models, which is incremental as it builds on existing CLIP methods with a domain-specific strategy.
The paper tackled the problem of limited sensitivity to fine-grained structural variations in diagram understanding for multimodal models, proposing a new training paradigm with pseudo contrastive samples that improved performance on image-text matching and visual question answering tasks over standard CLIP and hard-negative CLIP training.
Recent multimodal models such as Contrastive Language-Image Pre-training (CLIP) have shown remarkable ability to align visual and linguistic representations. However, domains where small visual differences carry large semantic significance, such as diagram understanding, remain challenging due to the models' limited sensitivity to fine-grained structural variations. We propose a new training paradigm designed to enhance diagram comprehension in vision-language models. Our approach introduces pseudo contrastive samples generated by a diagram renderer that creates synthetic diagrams using randomly picked text elements. These samples highlight structural differences in diagrammatic imagery without requiring any modification or editing of the original data. By incorporating these pseudo contrastive samples into the training objective, the model learns to capture more precise and semantically consistent diagram structures. Empirical evaluations on a benchmark dataset of flowcharts demonstrate substantial improvements over standard CLIP and hard-negative CLIP training in both image-text matching and visual question answering tasks. The results underscore the value of domain-specific training strategies and contribute to advancing diagrammatic understanding within the broader context of vision-language learning.