Diversifying Joint Vision-Language Tokenization Learning
This work addresses the need for better generalization in multimodal AI tasks like Visual Question Answering, though it appears incremental as it builds on existing joint representation methods.
The paper tackles the problem of improving generalization in joint vision-language representations by diversifying tokenization learning, and it reports that the approach outperforms baseline models in most settings and is competitive with state-of-the-art methods.
Building joint representations across images and text is an essential step for tasks such as Visual Question Answering and Video Question Answering. In this work, we find that the representations must not only jointly capture features from both modalities but should also be diverse for better generalization performance. To this end, we propose joint vision-language representation learning by diversifying the tokenization learning process, enabling tokens that are sufficiently disentangled from each other to be learned from both modalities. We observe that our approach outperforms the baseline models in a majority of settings and is competitive with state-of-the-art methods.