When the Sun Goes Down: Repairing Photometric Losses for All-Day Depth Estimation
This addresses the limitation of existing methods for autonomous vehicles and robotics that require reliable depth estimation in all-day scenarios, though it is incremental as it builds on established photometric loss frameworks.
The paper tackles the problem of self-supervised depth and ego-motion estimation degrading under nighttime conditions due to photometric loss violations, and shows that combining neural intensity transformation, residual flow correction, and image denoising enables a single model to work for both day and night sequences on the Oxford RobotCar dataset.
Self-supervised deep learning methods for joint depth and ego-motion estimation can yield accurate trajectories without needing ground-truth training data. However, as they typically use photometric losses, their performance can degrade significantly when the assumptions these losses make (e.g. temporal illumination consistency, a static scene, and the absence of noise and occlusions) are violated. This limits their use for e.g. nighttime sequences, which tend to contain many point light sources (including on dynamic objects) and low signal-to-noise ratio (SNR) in darker image regions. In this paper, we show how to use a combination of three techniques to allow the existing photometric losses to work for both day and nighttime images. First, we introduce a per-pixel neural intensity transformation to compensate for the light changes that occur between successive frames. Second, we predict a per-pixel residual flow map that we use to correct the reprojection correspondences induced by the estimated ego-motion and depth from the networks. And third, we denoise the training images to improve the robustness and accuracy of our approach. These changes allow us to train a single model for both day and nighttime images without needing separate encoders or extra feature networks like existing methods. We perform extensive experiments and ablation studies on the challenging Oxford RobotCar dataset to demonstrate the efficacy of our approach for both day and nighttime sequences.