Worth Remembering: Surprise-Gated Robot Episodic Memory
This work addresses the problem of long-term memory selection for generalist robots, enabling them to ground instructions in past experiences without storing all events.
The paper proposes using Bayesian surprise as a gating mechanism for selective episodic memory in robots, enabling efficient storage of high-utility past events without prior knowledge of future tasks. The method achieves ≥12% improvement over prior robot memory methods in question answering and surpasses supervised methods in event segmentation.
Robots solving generalist tasks need to be able to ground instructions in their past experience, since humans may refer to notable past events when giving a task (e.g., ``Take me to where the chemical spill happened yesterday''). Since memory limits make storing all past events infeasible, long-term robot memory must be selective, ideally retaining only those episodes with high utility for future tasks. However, future tasks are not typically given a priori for generalist robots. To select generically useful memories, we propose Bayesian surprise as a gating mechanism for memory formation. We present an approach to compute surprise in a semantically rich deployment-agnostic latent space provided by V-JEPA-2. Using our gated episodic memory to augment 4D scene graph-based spatial memory, we show a consistent improvement over state-of-the-art benchmarks in robot question answering, outperforming prior robot memory methods by $\geq12\%$ for temporal, spatial, and binary questions, and surpassing the performance of supervised and non-causal methods with an unsupervised causal method in event segmentation tasks.