Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline
This work addresses visual dialog for AI systems, but it is incremental as it adapts an existing method to a new task with specific gains.
The authors tackled the problem of improving visual dialog performance by leveraging pretraining on related vision-language datasets before transferring to VisDial, resulting in a model that outperforms prior work by more than 1% on NDCG and MRR and highlighting a trade-off between metrics when using dense annotations.
Prior work in visual dialog has focused on training deep neural models on VisDial in isolation. Instead, we present an approach to leverage pretraining on related vision-language datasets before transferring to visual dialog. We adapt the recently proposed ViLBERT (Lu et al., 2019) model for multi-turn visually-grounded conversations. Our model is pretrained on the Conceptual Captions and Visual Question Answering datasets, and finetuned on VisDial. Our best single model outperforms prior published work (including model ensembles) by more than 1% absolute on NDCG and MRR. Next, we find that additional finetuning using "dense" annotations in VisDial leads to even higher NDCG -- more than 10% over our base model -- but hurts MRR -- more than 17% below our base model! This highlights a trade-off between the two primary metrics -- NDCG and MRR -- which we find is due to dense annotations not correlating well with the original ground-truth answers to questions.