Pre-training image-language transformers for open-vocabulary tasks
This work addresses the challenge of improving performance on text-generative vision-language tasks for AI applications, though it appears incremental as it builds on existing pre-training methods.
The authors tackled the problem of pre-training vision-language transformers for open-vocabulary tasks by using a mixture of tasks including image-text captioning and object-aware strategies, resulting in large gains over standard methods on tasks like Visual Question Answering, visual entailment, and captioning.
We present a pre-training approach for vision and language transformer models, which is based on a mixture of diverse tasks. We explore both the use of image-text captioning data in pre-training, which does not need additional supervision, as well as object-aware strategies to pre-train the model. We evaluate the method on a number of textgenerative vision+language tasks, such as Visual Question Answering, visual entailment and captioning, and demonstrate large gains over standard pre-training methods.