MTVR: Multilingual Moment Retrieval in Videos
This work addresses the need for multilingual video retrieval datasets and models, though it is incremental as it extends an existing English dataset to include Chinese.
The authors tackled the problem of multilingual video moment retrieval by introducing mTVR, a large-scale dataset with 218K English and Chinese queries from 21.8K TV show clips, and proposed mXML, a model that outperforms monolingual baselines while using fewer parameters.
We introduce mTVR, a large-scale multilingual video moment retrieval dataset, containing 218K English and Chinese queries from 21.8K TV show video clips. The dataset is collected by extending the popular TVR dataset (in English) with paired Chinese queries and subtitles. Compared to existing moment retrieval datasets, mTVR is multilingual, larger, and comes with diverse annotations. We further propose mXML, a multilingual moment retrieval model that learns and operates on data from both languages, via encoder parameter sharing and language neighborhood constraints. We demonstrate the effectiveness of mXML on the newly collected MTVR dataset, where mXML outperforms strong monolingual baselines while using fewer parameters. In addition, we also provide detailed dataset analyses and model ablations. Data and code are publicly available at https://github.com/jayleicn/mTVRetrieval