SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation
This work addresses the challenge of long-horizon reasoning in mobile manipulation for robotics, providing a method that enhances policy performance in household environments.
The paper introduces SERF, a spatiotemporal feature map that improves long-horizon mobile manipulation by conditioning a vision-language-action policy on a shared latent representation of the environment and robot body. The SERF VLA policy outperforms image-only baselines on BEHAVIOR-1K, achieving faster subgoal completion, better robustness to scene changes, and recovery from object-drop failures.
Long-horizon robot mobile manipulation requires continual reasoning about localization, environment changes, and task progress, all of which are challenging to infer from image observations alone. In this paper, we show that conditioning a mobile manipulation policy on a spatiotemporal feature map improves reasoning over long horizons. The map represents the environment and the articulated robot body as neural points in a shared latent space and is updated online from egocentric observations and proprioceptive state. We update the environment neural points using object-level rigid tracking and the robot neural points using forward kinematics. We use our spatiotemporal environment and robot feature (SERF) map as a state input to a vision-language-action (VLA) model by extracting map tokens from multiple reference frames and spatial scales, providing the policy with both local and global context. We demonstrate SERF on BEHAVIOR-1K, a benchmark for long-horizon mobile manipulation in household environments. Experiments show that the SERF VLA policy outperforms image-only baselines, reaches subgoals faster by following more direct trajectories, improves robustness to scene-configuration shifts, and recovers from object-drop failures.