HippoMM: Hippocampal-inspired Multimodal Memory for Long Audiovisual Event Understanding
This addresses the problem of temporal integration and cross-modal associations in computational systems for researchers in multimodal AI, though it appears incremental as it builds on known hippocampal mechanisms.
The paper tackled the challenge of long audiovisual event understanding by introducing HippoMM, a hippocampal-inspired multimodal memory architecture, achieving state-of-the-art 78.2% accuracy on the HippoVlog benchmark while operating 5x faster than baselines.
Comprehending extended audiovisual experiences remains challenging for computational systems, particularly temporal integration and cross-modal associations fundamental to human episodic memory. We introduce HippoMM, a computational cognitive architecture that maps hippocampal mechanisms to solve these challenges. Rather than relying on scaling or architectural sophistication, HippoMM implements three integrated components: (i) Episodic Segmentation detects audiovisual input changes to split videos into discrete episodes, mirroring dentate gyrus pattern separation; (ii) Memory Consolidation compresses episodes into summaries with key features preserved, analogous to hippocampal memory formation; and (iii) Hierarchical Memory Retrieval first searches semantic summaries, then escalates via temporal window expansion around seed segments for cross-modal queries, mimicking CA3 pattern completion. These components jointly create an integrated system exceeding the sum of its parts. On our HippoVlog benchmark testing associative memory, HippoMM achieves state-of-the-art 78.2% accuracy while operating 5x faster than retrieval-augmented baselines. Our results demonstrate that cognitive architectures provide blueprints for next-generation multimodal understanding. The code and benchmark dataset are publicly available at https://github.com/linyueqian/HippoMM.