Principles of Visual Tokens for Efficient Video Understanding
This work addresses efficiency challenges in video understanding for AI applications, offering a novel method that outperforms existing baselines, though it builds on incremental insights from token selection research.
The paper tackled the problem of inefficient video understanding by analyzing visual token principles and proposing LITE, a lightweight model that selects a small number of tokens, achieving state-of-the-art performance with improved computation-accuracy trade-offs on datasets like Kinetics-400 and Something-Something-V2.
Video understanding has made huge strides in recent years, relying largely on the power of transformers. As this architecture is notoriously expensive and video data is highly redundant, research into improving efficiency has become particularly relevant. Some creative solutions include token selection and merging. While most methods succeed in reducing the cost of the model and maintaining accuracy, an interesting pattern arises: most methods do not outperform the baseline of randomly discarding tokens. In this paper we take a closer look at this phenomenon and observe 5 principles of the nature of visual tokens. For example, we observe that the value of tokens follows a clear Pareto-distribution where most tokens have remarkably low value, and just a few carry most of the perceptual information. We build on these and further insights to propose a lightweight video model, LITE, that can select a small number of tokens effectively, outperforming state-of-the-art and existing baselines across datasets (Kinetics-400 and Something-Something-V2) in the challenging trade-off of computation (GFLOPs) vs accuracy. Experiments also show that LITE generalizes across datasets and even other tasks without the need for retraining.