Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning
This provides a scalable solution for joint image and video learning, benefiting researchers and practitioners in computer vision, though it is incremental as it builds on existing ViT methods.
The paper tackles the problem of adapting Vision Transformer (ViT) encoders to efficiently handle both image and video inputs by using sparse sampling, achieving state-of-the-art results without requiring full fine-tuning.
We present a simple approach which can turn a ViT encoder into an efficient video model, which can seamlessly work with both image and video inputs. By sparsely sampling the inputs, the model is able to do training and inference from both inputs. The model is easily scalable and can be adapted to large-scale pre-trained ViTs without requiring full finetuning. The model achieves SOTA results and the code will be open-sourced.