A Visual Tour Of Current Challenges In Multimodal Language Models
This work addresses a challenge in multimodal learning for NLP researchers, but it is incremental as it highlights limitations without proposing new solutions.
The study investigated whether visual grounding in multimodal language models helps acquire function words, finding that stable diffusion models effectively model only a few pronoun subcategories and relatives out of seven categories.
Transformer models trained on massive text corpora have become the de facto models for a wide range of natural language processing tasks. However, learning effective word representations for function words remains challenging. Multimodal learning, which visually grounds transformer models in imagery, can overcome the challenges to some extent; however, there is still much work to be done. In this study, we explore the extent to which visual grounding facilitates the acquisition of function words using stable diffusion models that employ multimodal models for text-to-image generation. Out of seven categories of function words, along with numerous subcategories, we find that stable diffusion models effectively model only a small fraction of function words -- a few pronoun subcategories and relatives. We hope that our findings will stimulate the development of new datasets and approaches that enable multimodal models to learn better representations of function words.