Long-range Multimodal Pretraining for Movie Understanding
This work addresses the need for improved multimodal models in movie understanding tasks, offering incremental advancements in data efficiency and performance.
The authors tackled the problem of movie understanding by introducing a long-range multimodal pretraining strategy that leverages movie data to train transferable encoders, achieving state-of-the-art results on several LVU tasks and setting new benchmarks in five different evaluations while being more data-efficient.
Learning computer vision models from (and for) movies has a long-standing history. While great progress has been attained, there is still a need for a pretrained multimodal model that can perform well in the ever-growing set of movie understanding tasks the community has been establishing. In this work, we introduce Long-range Multimodal Pretraining, a strategy, and a model that leverages movie data to train transferable multimodal and cross-modal encoders. Our key idea is to learn from all modalities in a movie by observing and extracting relationships over a long-range. After pretraining, we run ablation studies on the LVU benchmark and validate our modeling choices and the importance of learning from long-range time spans. Our model achieves state-of-the-art on several LVU tasks while being much more data efficient than previous works. Finally, we evaluate our model's transferability by setting a new state-of-the-art in five different benchmarks.