VidCompress: Memory-Enhanced Temporal Compression for Video Understanding in Large Language Models
This addresses video understanding challenges in multimodal large language models, offering an incremental improvement over prior methods.
The paper tackles the problem of insufficient temporal-spatial interaction and limited visual token capacity in Video-LLMs for video understanding by proposing VidCompress, a memory-enhanced temporal compression method, which significantly outperforms existing models on VideoQA datasets and benchmarks.
Video-based multimodal large language models (Video-LLMs) possess significant potential for video understanding tasks. However, most Video-LLMs treat videos as a sequential set of individual frames, which results in insufficient temporal-spatial interaction that hinders fine-grained comprehension and difficulty in processing longer videos due to limited visual token capacity. To address these challenges, we propose VidCompress, a novel Video-LLM featuring memory-enhanced temporal compression. VidCompress employs a dual-compressor approach: a memory-enhanced compressor captures both short-term and long-term temporal relationships in videos and compresses the visual tokens using a multiscale transformer with a memory-cache mechanism, while a text-perceived compressor generates condensed visual tokens by utilizing Q-Former and integrating temporal contexts into query embeddings with cross attention. Experiments on several VideoQA datasets and comprehensive benchmarks demonstrate that VidCompress efficiently models complex temporal-spatial relations and significantly outperforms existing Video-LLMs.