MDMMT: Multidomain Multimodal Transformer for Video Retrieval
This addresses video retrieval for AI applications, but it is incremental as it builds on existing transformer and multimodal methods.
The paper tackled text-to-video retrieval by introducing a multidomain multimodal transformer, achieving state-of-the-art results on MSRVTT and LSMDC benchmarks with a single model without finetuning, and showing that training on different datasets improves test performance.
We present a new state-of-the-art on the text to video retrieval task on MSRVTT and LSMDC benchmarks where our model outperforms all previous solutions by a large margin. Moreover, state-of-the-art results are achieved with a single model on two datasets without finetuning. This multidomain generalisation is achieved by a proper combination of different video caption datasets. We show that training on different datasets can improve test results of each other. Additionally we check intersection between many popular datasets and found that MSRVTT has a significant overlap between the test and the train parts, and the same situation is observed for ActivityNet.