Mem4D: Decoupling Static and Dynamic Memory for Dynamic Scene Reconstruction
This work solves the challenge of accurate and efficient dynamic scene reconstruction for applications like robotics and AR/VR, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackles the problem of reconstructing dense geometry for dynamic scenes from monocular video by addressing the Memory Demand Dilemma, where existing methods compromise between static stability and dynamic detail, and proposes Mem4D, which decouples static and dynamic memory to achieve state-of-the-art or competitive performance with high efficiency.
Reconstructing dense geometry for dynamic scenes from a monocular video is a critical yet challenging task. Recent memory-based methods enable efficient online reconstruction, but they fundamentally suffer from a Memory Demand Dilemma: The memory representation faces an inherent conflict between the long-term stability required for static structures and the rapid, high-fidelity detail retention needed for dynamic motion. This conflict forces existing methods into a compromise, leading to either geometric drift in static structures or blurred, inaccurate reconstructions of dynamic objects. To address this dilemma, we propose Mem4D, a novel framework that decouples the modeling of static geometry and dynamic motion. Guided by this insight, we design a dual-memory architecture: 1) The Transient Dynamics Memory (TDM) focuses on capturing high-frequency motion details from recent frames, enabling accurate and fine-grained modeling of dynamic content; 2) The Persistent Structure Memory (PSM) compresses and preserves long-term spatial information, ensuring global consistency and drift-free reconstruction for static elements. By alternating queries to these specialized memories, Mem4D simultaneously maintains static geometry with global consistency and reconstructs dynamic elements with high fidelity. Experiments on challenging benchmarks demonstrate that our method achieves state-of-the-art or competitive performance while maintaining high efficiency. Codes will be publicly available.