Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs
This meta-analysis provides clarity for researchers and practitioners working with multimodal pretraining, helping them understand the underlying mechanisms and performance factors of V&L BERTs.
This paper analyzes existing vision-and-language BERT models, unifying single-stream and dual-stream encoders under a single theoretical framework. Through controlled experiments on five V&L BERTs, it reveals that training data and hyperparameters are the primary drivers of performance differences, with the embedding layer also playing a crucial role.
Large-scale pretraining and task-specific fine-tuning is now the standard methodology for many tasks in computer vision and natural language processing. Recently, a multitude of methods have been proposed for pretraining vision and language BERTs to tackle challenges at the intersection of these two key areas of AI. These models can be categorised into either single-stream or dual-stream encoders. We study the differences between these two categories, and show how they can be unified under a single theoretical framework. We then conduct controlled experiments to discern the empirical differences between five V&L BERTs. Our experiments show that training data and hyperparameters are responsible for most of the differences between the reported results, but they also reveal that the embedding layer plays a crucial role in these massive models.