Rolling-Shutter-Aware Differential SfM and Image Rectification
This addresses image distortion issues for applications in computer vision and photography, but it is incremental as it builds on existing SfM methods with specific modifications for rolling shutter.
The paper tackles the problem of rolling shutter artifacts in cameras by developing a modified differential Structure from Motion algorithm that estimates relative pose and rectifies images, showing improved accuracy in pose estimation and 3D reconstruction, and outperforming commercial software in artifact removal.
In this paper, we develop a modified differential Structure from Motion (SfM) algorithm that can estimate relative pose from two consecutive frames despite of Rolling Shutter (RS) artifacts. In particular, we show that under constant velocity assumption, the errors induced by the rolling shutter effect can be easily rectified by a linear scaling operation on each optical flow. We further propose a 9-point algorithm to recover the relative pose of a rolling shutter camera that undergoes constant acceleration motion. We demonstrate that the dense depth maps recovered from the relative pose of the RS camera can be used in a RS-aware warping for image rectification to recover high-quality Global Shutter (GS) images. Experiments on both synthetic and real RS images show that our RS-aware differential SfM algorithm produces more accurate results on relative pose estimation and 3D reconstruction from images distorted by RS effect compared to standard SfM algorithms that assume a GS camera model. We also demonstrate that our RS-aware warping for image rectification method outperforms state-of-the-art commercial software products, i.e. Adobe After Effects and Apple Imovie, at removing RS artifacts.