IMU-Assisted Learning of Single-View Rolling Shutter Correction
This addresses the problem of rolling shutter distortion for photography and computer vision algorithms like visual SLAM, enabling more accurate results with rolling shutter cameras, though it is incremental as it builds on existing methods by adding IMU data.
The paper tackles rolling shutter distortion in images by proposing a deep neural network that predicts depth and row-wise pose from a single image, incorporating IMU data to enhance pose prediction, resulting in marked improvements for the DSO algorithm when using corrected imagery.
Rolling shutter distortion is highly undesirable for photography and computer vision algorithms (e.g., visual SLAM) because pixels can be potentially captured at different times and poses. In this paper, we propose a deep neural network to predict depth and row-wise pose from a single image for rolling shutter correction. Our contribution in this work is to incorporate inertial measurement unit (IMU) data into the pose refinement process, which, compared to the state-of-the-art, greatly enhances the pose prediction. The improved accuracy and robustness make it possible for numerous vision algorithms to use imagery captured by rolling shutter cameras and produce highly accurate results. We also extend a dataset to have real rolling shutter images, IMU data, depth maps, camera poses, and corresponding global shutter images for rolling shutter correction training. We demonstrate the efficacy of the proposed method by evaluating the performance of Direct Sparse Odometry (DSO) algorithm on rolling shutter imagery corrected using the proposed approach. Results show marked improvements of the DSO algorithm over using uncorrected imagery, validating the proposed approach.