RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies
This benchmark addresses the lack of standardized evaluation for memory mechanisms in VLA models, which is crucial for advancing robotic generalist policies.
This paper introduces RoboMME, a large-scale benchmark with 16 manipulation tasks to evaluate vision-language-action (VLA) models on long-horizon and history-dependent robotic tasks. The study also develops 14 memory-augmented VLA variants, revealing that memory representation effectiveness is highly task-dependent.
Memory is critical for long-horizon and history-dependent robotic manipulation. Such tasks often involve counting repeated actions or manipulating objects that become temporarily occluded. Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms; however, their evaluations remain confined to narrow, non-standardized settings. This limits their systematic understanding, comparison, and progress measurement. To address these challenges, we introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models in long-horizon, history-dependent scenarios. Our benchmark comprises 16 manipulation tasks constructed under a carefully designed taxonomy that evaluates temporal, spatial, object, and procedural memory. We further develop a suite of 14 memory-augmented VLA variants built on the π0.5 backbone to systematically explore different memory representations across multiple integration strategies. Experimental results show that the effectiveness of memory representations is highly task-dependent, with each design offering distinct advantages and limitations across different tasks. Videos and code can be found at our website https://robomme.github.io.