Learning Physical Principles from Interaction: Self-Evolving Planning via Test-Time Memory
This addresses the challenge of reliable object manipulation for robots by enabling adaptive learning from experience, though it is incremental as it builds on existing VLM planners.
The paper tackles the problem of VLM robot planners lacking specific physical predictions for object manipulation by introducing PhysMem, a memory framework that learns physical principles from test-time interaction without parameter updates, achieving 76% success on a brick insertion task compared to 23% for direct retrieval.
Reliable object manipulation requires understanding physical properties that vary across objects and environments. Vision-language model (VLM) planners can reason about friction and stability in general terms; however, they often cannot predict how a specific ball will roll on a particular surface or which stone will provide a stable foundation without direct experience. We present PhysMem, a memory framework that enables VLM robot planners to learn physical principles from interaction at test time, without updating model parameters. The system records experiences, generates candidate hypotheses, and verifies them through targeted interaction before promoting validated knowledge to guide future decisions. A central design choice is verification before application: the system tests hypotheses against new observations rather than applying retrieved experience directly, reducing rigid reliance on prior experience when physical conditions change. We evaluate PhysMem on three real-world manipulation tasks and simulation benchmarks across four VLM backbones. On a controlled brick insertion task, principled abstraction achieves 76% success compared to 23% for direct experience retrieval, and real-world experiments show consistent improvement over 30-minute deployment sessions.