Frequency Is What You Need: Word-frequency Masking Benefits Vision-Language Model Pre-training
This work addresses the challenge of reducing training set sizes for Vision-Language Models, presenting an incremental improvement in masking strategies.
The paper tackles the problem of inefficient training of Vision-Language Models by analyzing text masking strategies, showing that syntax masking is outperformed by other approaches with sufficient epochs and that their proposed frequency-based method (CLIPF) offers advantages, especially with fewer input tokens.
Vision Language Models (VLMs) can be trained more efficiently if training sets can be reduced in size. Recent work has shown the benefits of masking text during VLM training using a variety of approaches: truncation, random masking, block masking and syntax masking. In this paper, we show that the best masking strategy changes over training epochs and that, given sufficient training epochs. We analyze existing text masking approaches including syntax masking, which is currently the state of the art, and identify the word frequency distribution as important in determining their success. Experiments on a large range of data sets demonstrate that syntax masking is outperformed by other approaches, given sufficient epochs, and that our proposed frequency-based approach, called Contrastive Language-Image Pre-training with Word Frequency Masking (CLIPF) has numerous advantages. The benefits are particularly evident as the number of input tokens decreases.