RenderMem: Rendering as Spatial Memory Retrieval
This addresses the challenge of geometric grounding in spatial reasoning for embodied AI agents, though it is incremental as it builds on existing vision-language models without architectural changes.
The paper tackles the problem of viewpoint-dependent reasoning in embodied agents by introducing RenderMem, a spatial memory framework that uses rendering to generate visual evidence from 3D scene representations, resulting in consistent improvements on visibility and occlusion queries in the AI2-THOR environment.
Embodied reasoning is inherently viewpoint-dependent: what is visible, occluded, or reachable depends critically on where the agent stands. However, existing spatial memory systems for embodied agents typically store either multi-view observations or object-centric abstractions, making it difficult to perform reasoning with explicit geometric grounding. We introduce RenderMem, a spatial memory framework that treats rendering as the interface between 3D world representations and spatial reasoning. Instead of storing fixed observations, RenderMem maintains a 3D scene representation and generates query-conditioned visual evidence by rendering the scene from viewpoints implied by the query. This enables embodied agents to reason directly about line-of-sight, visibility, and occlusion from arbitrary perspectives. RenderMem is fully compatible with existing vision-language models and requires no modification to standard architectures. Experiments in the AI2-THOR environment show consistent improvements on viewpoint-dependent visibility and occlusion queries over prior memory baselines.