Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling
This addresses efficient multimodal learning for low-resource scenarios, though it's incremental with constraints from the BabyLM Challenge.
The authors tackled the problem of training vision-language models with limited data by proposing a lightweight decoder-based architecture with token-wise dynamic gating, achieving competitive or superior performance on five benchmarks including BLiMP and VQA.
Training vision-language models on cognitively-plausible amounts of data requires rethinking how models integrate multimodal information. Within the constraints of the Vision track for the BabyLM Challenge 2025, we propose a lightweight decoder-based architecture with (1) token-wise dynamic gating for adaptive fusion of linguistic and visual cues, (2) feature modulation and channel attention to maximise the utility of limited visual information and (3) auxiliary contrastive objectives for visual grounding. Evaluation on five benchmarks (BLiMP, BLiMP Supplement, EWoK, Winoground and VQA) shows competitive or superior performance to multimodal baselines. More notably, our dynamic gate discovers interpretable patterns without explicit supervision, favouring visual cues for content words and linguistic cues for function words. While we identify limitations in the Challenge constraints, such as the information bottleneck created by global image embeddings and training instability from the dataset split, our findings establish dynamic gating as a powerful tool for efficient multimodal learning, offering both interpretability and performance even under severe constraints.