FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filtering
This addresses inefficiencies in video understanding for users of multi-modal LLMs, though it is incremental as it builds on existing methods.
The paper tackles the problem of redundant computation and visual noise in long, untrimmed videos by proposing FocusChat, a text-guided multi-modal LLM that filters spatiotemporal information to align visual tokens with user queries, achieving significant performance gains with fewer visual tokens and less training data.
Recently, multi-modal large language models have made significant progress. However, visual information lacking of guidance from the user's intention may lead to redundant computation and involve unnecessary visual noise, especially in long, untrimmed videos. To address this issue, we propose FocusChat, a text-guided multi-modal large language model (LLM) that emphasizes visual information correlated to the user's prompt. In detail, Our model first undergoes the semantic extraction module, which comprises a visual semantic branch and a text semantic branch to extract image and text semantics, respectively. The two branches are combined using the Spatial-Temporal Filtering Module (STFM). STFM enables explicit spatial-level information filtering and implicit temporal-level feature filtering, ensuring that the visual tokens are closely aligned with the user's query. It lowers the essential number of visual tokens inputted into the LLM. FocusChat significantly outperforms Video-LLaMA in zero-shot experiments, using an order of magnitude less training data with only 16 visual tokens occupied. It achieves results comparable to the state-of-the-art in few-shot experiments, with only 0.72M pre-training data.