MUT3R: Motion-aware Updating Transformer for Dynamic 3D Reconstruction
This addresses motion artifacts in streaming 3D reconstruction for computer vision applications, but is incremental as it builds on existing transformer architectures without retraining.
The paper tackles motion-induced artifacts in dynamic 3D reconstruction by introducing MUT3R, a training-free framework that uses attention-derived motion cues to suppress dynamic content in transformer layers, improving temporal consistency and camera pose robustness across benchmarks.
Recent stateful recurrent neural networks have achieved remarkable progress on static 3D reconstruction but remain vulnerable to motion-induced artifacts, where non-rigid regions corrupt attention propagation between the spatial memory and image feature. By analyzing the internal behaviors of the state and image token updating mechanism, we find that aggregating self-attention maps across layers reveals a consistent pattern: dynamic regions are naturally down-weighted, exposing an implicit motion cue that the pretrained transformer already encodes but never explicitly uses. Motivated by this observation, we introduce MUT3R, a training-free framework that applies the attention-derived motion cue to suppress dynamic content in the early layers of the transformer during inference. Our attention-level gating module suppresses the influence of dynamic regions before their artifacts propagate through the feature hierarchy. Notably, we do not retrain or fine-tune the model; we let the pretrained transformer diagnose its own motion cues and correct itself. This early regulation stabilizes geometric reasoning in streaming scenarios and leads to improvements in temporal consistency and camera pose robustness across multiple dynamic benchmarks, offering a simple and training-free pathway toward motion-aware streaming reconstruction.