Fine-grained Multi-Modal Self-Supervised Learning
This addresses the computational bottleneck for researchers and practitioners in multi-modal learning, though it is incremental as it builds on existing self-supervised methods.
The paper tackles the problem of noisy and computationally expensive multi-modal self-supervised learning from videos by proposing a fine-grained training scheme that uses attention to reduce noise, enabling smaller models and batch sizes to achieve performance comparable to state-of-the-art on tasks like action recognition and text-image retrieval.
Multi-Modal Self-Supervised Learning from videos has been shown to improve model's performance on various downstream tasks. However, such Self-Supervised pre-training requires large batch sizes and a large amount of computation resources due to the noise present in the uncurated data. This is partly due to the fact that the prevalent training scheme is trained on coarse-grained setting, in which vectors representing the whole video clips or natural language sentences are used for computing similarity. Such scheme makes training noisy as part of the video clips can be totally not correlated with the other-modality input such as text description. In this paper, we propose a fine-grained multi-modal self-supervised training scheme that computes the similarity between embeddings at finer-scale (such as individual feature map embeddings and embeddings of phrases), and uses attention mechanisms to reduce noisy pairs' weighting in the loss function. We show that with the proposed pre-training scheme, we can train smaller models, with smaller batch-size and much less computational resources to achieve downstream tasks performances comparable to State-Of-The-Art, for tasks including action recognition and text-image retrievals.