ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction
For researchers in 3D reconstruction, this method improves long-sequence performance without retraining, though it is an incremental improvement over existing recurrent architectures.
ReCal3R introduces a reliability-calibrated learning rate method for recurrent 3D reconstruction that prevents state corruption from noisy observations, achieving a 3.7× reduction in ATE on long sequences with comparable runtime and memory.
Streaming 3D reconstruction relies on a compact recurrent scene state to process long image streams in linear time and bounded memory. However, repeated updates can gradually corrupt this state, causing reliable historical information to be overwritten by noisy or ambiguous observations. We introduce ReCal3R, a reliability-calibrated learning rate method for recurrent 3D reconstruction. Instead of directly applying a candidate learning rate, our method estimates state token reliability from the maintained scene state and uses it to calibrate a candidate learning rate derived from token alignment, state reconstruction residual, and recent update pressure. The resulting token-wise learning rate interpolates between a conservative base rate and the candidate rate, suppressing aggressive updates on unreliable tokens while preserving adaptation to informative frames. Applied to CUT3R as a training-free calibration rule, ReCal3R reaches strong performance on long sequences in pose, depth, and reconstruction quality, including a 3.7$\times$ reduction in ATE, with comparable runtime and memory. Code is available at: https://github.com/Powertony102/ReCal3R.