RT-Cache: Training-Free Retrieval for Real-Time Manipulation
This addresses the challenge of heavy per-step inference or deployment-time fine-tuning for real-time robot manipulation, offering a scalable solution for few-shot deployment.
The paper tackles the problem of enabling real robots to repeat behaviors in new environments with minimal new data, proposing RT-Cache, a training-free retrieval-as-control pipeline that caches image-action trajectories for real-time control, achieving approximately 2x higher success and 30% faster completion times than baselines.
Real robots are expected to repeat the same behavior in new environments with very little new data, yet modern controllers either incur heavy per-step inference or require deployment-time fine-tuning. We propose RT-Cache, a training-free retrieval-as-control pipeline that caches diverse image action trajectories in a unified vector memory and, at test time, embeds the current frame to retrieve and replay multi-step snippets, replacing per-step model calls. A hierarchical search keeps lookups sub-second at million scale, shifting cost from compute to storage and enabling real-time control on modest GPUs. Across real-robot tasks and large open logs, RT-Cache achieves higher success and lower completion time than strong retrieval baselines (approximately x2 higher success and ~30% faster in our settings), and a single-episode anchoring study shows immediate adaptation to a more complex, contact-rich task without fine-tuning. RT-Cache turns experience into an append-only memory, offering a simple, scalable path to few-shot deployment today and a foundation for multimodal keys and optional integration with high-level policies. Project page: https://rt-cache.github.io/.