Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding
This addresses the problem of memory-efficient real-time video understanding for applications requiring procedural analysis, though it appears incremental as it builds on existing multimodal LLM approaches with a novel caching mechanism.
The authors tackled real-time procedural video understanding by introducing ProVideLLM, an end-to-end framework that reduces token count by 22x for long-term observations and achieves streaming inference at 10 FPS and dialogue at 25 FPS with a 2GB GPU memory footprint, while setting new state-of-the-art results on six tasks across four datasets.
We introduce ProVideLLM, an end-to-end framework for real-time procedural video understanding. ProVideLLM integrates a multimodal cache configured to store two types of tokens - verbalized text tokens, which provide compressed textual summaries of long-term observations, and visual tokens, encoded with DETR-QFormer to capture fine-grained details from short-term observations. This design reduces token count by 22x over existing methods in representing one hour of long-term observations while effectively encoding fine-granularity of the present. By interleaving these tokens in our multimodal cache, ProVideLLM ensures sub-linear scaling of memory and compute with video length, enabling per-frame streaming inference at 10 FPS and streaming dialogue at 25 FPS, with a minimal 2GB GPU memory footprint. ProVideLLM also sets new state-of-the-art results on six procedural tasks across four datasets.