Beyond Retrieval: Analytic Memory for Multimodal Agents
This work addresses the limitation of existing multimodal memory systems for agents that primarily focus on retrieval, by introducing a complementary analytic memory component to enable more complex computations over accumulated observations.
This paper introduces analytic memory, a new abstraction for multimodal agents that organizes recurring observations into queryable structures for filtering, aggregation, ranking, and temporal comparison. The proposed framework, AdaMM, jointly supports retrieval and analytic memory, improving performance on MemEye by up to 11.3% and on MemGallery by up to 7.3%.
Long-term multimodal memory must support not only retrieving relevant information but also computing over observations accumulated across interactions. Existing systems largely emphasize \emph{retrieval memory}, organizing interaction histories through summaries and indexes to return query-relevant information at multiple granularities, from high-level abstractions to underlying records. In this paper, we formulate \emph{analytic memory} as a complementary abstraction that organizes recurring multimodal observations into queryable structures supporting filtering, aggregation, ranking, and temporal comparison. We present AdaMM, a framework that jointly supports retrieval and analytic memory. Rather than relying on application-defined schemas, AdaMM extracts provenance-linked attribute-value observations from dialogue, images, and contextual metadata, discovers recurring field structures, and materializes them for analytical access. At inference time, a memory-aware planner decomposes queries into retrieval and analytic operations and routes each operation to the appropriate tools. Experiments on two long-term multimodal memory benchmarks, MemEye and MemGallery, show that AdaMM improves performance by up to 11.3\% and 7.3\%, respectively.