Open-ended Hierarchical Streaming Video Understanding with Vision Language Models
This work addresses video understanding for AI systems by enabling more detailed temporal analysis, though it appears incremental as it builds on existing vision-language models and datasets.
The paper tackles the problem of hierarchical streaming video understanding by combining online temporal action localization with free-form description generation, proposing OpenHOUSE which nearly doubles performance for detecting boundaries between closely adjacent actions compared to direct extensions of existing methods.
We introduce Hierarchical Streaming Video Understanding, a task that combines online temporal action localization with free-form description generation. Given the scarcity of datasets with hierarchical and fine-grained temporal annotations, we demonstrate that LLMs can effectively group atomic actions into higher-level events, enriching existing datasets. We then propose OpenHOUSE (Open-ended Hierarchical Online Understanding System for Events), which extends streaming action perception beyond action classification. OpenHOUSE features a specialized streaming module that accurately detects boundaries between closely adjacent actions, nearly doubling the performance of direct extensions of existing methods. We envision the future of streaming action perception in the integration of powerful generative models, with OpenHOUSE representing a key step in that direction.