Centered Masking for Language-Image Pre-Training
This work addresses the challenge of efficient and effective pre-training for vision-language models, offering an incremental improvement over existing methods like FLIP.
The paper tackles the problem of improving vision-language pre-training by introducing Gaussian masking (GLIP), which replaces random masking with centered masking based on a Gaussian distribution, resulting in improved performance across downstream datasets and tasks while maintaining computational savings.
We introduce Gaussian masking for Language-Image Pre-Training (GLIP) a novel, straightforward, and effective technique for masking image patches during pre-training of a vision-language model. GLIP builds on Fast Language-Image Pre-Training (FLIP), which randomly masks image patches while training a CLIP model. GLIP replaces random masking with centered masking, that uses a Gaussian distribution and is inspired by the importance of image patches at the center of the image. GLIP retains the same computational savings as FLIP, while improving performance across a range of downstream datasets and tasks, as demonstrated by our experimental results. We show the benefits of GLIP to be easy to obtain, requiring no delicate tuning of the Gaussian, and also applicable to data sets containing images without an obvious center focus.