FOLIO: Focused Semantic Memory for Streaming Video Understanding
This work addresses the memory bottleneck in online video understanding for systems that must handle unbounded streams and unpredictable queries, offering a practical solution that improves both accuracy and efficiency.
FOLIO introduces a training-free focused semantic memory system for streaming video understanding that selectively retains high-detail information for important entities while compressing surrounding context, achieving state-of-the-art performance (82.0/69.1 Perception/Backward accuracy on OVO-Bench and 74.5 overall accuracy on StreamingBench) with reduced memory cost.
In online streaming video understanding, a video stream continues to arrive and queries may be issued at any time. Because streaming frames grow without bound, the system must continuously compress and retain information from the observed video prefix while future frames and future queries remain unknown. The core challenge is deciding what information to retain and how to organize the maintained history: as this history grows with the stream, memory cost increases and many redundant visual details are retained, whereas later queries often depend on specific entities, actions, and their temporal changes. To address this challenge, we introduce FOLIO, a training-free focused semantic memory system that records important parts of the stream in higher detail while keeping surrounding context compact. As the stream arrives, FOLIO updates memory at the segment level, guided by a dynamic focus state, combining a short-term visual buffer with a long-term semantic memory organized around observed entities and linked to a visual-evidence cache. At query time, lightweight hybrid retrieval combines direct matching over the structured memory with semantic query expansion. FOLIO achieves state-of-the-art performance, reaching 82.0/69.1 Perception/Backward accuracy on OVO-Bench with Qwen3-VL-8B and 74.5 overall accuracy on StreamingBench, while substantially reducing the cost of maintaining streaming memory by reserving detailed records for focused entities and storing surrounding context compactly.