DeepOIS: Gyroscope-Guided Deep Optical Image Stabilizer Compensation
This addresses a practical issue for mobile photography and computer vision applications by enabling robust gyroscope-based alignment on OIS-equipped devices, though it is an incremental improvement over existing alignment techniques.
The paper tackles the problem of using gyroscope sensors for image alignment on cameras with optical image stabilizers (OIS), which typically disrupt this process, by proposing a deep network that compensates for OIS motions. The result shows that their method achieves performance comparable to non-OIS cameras and outperforms image-based alignment by a significant margin.
Mobile captured images can be aligned using their gyroscope sensors. Optical image stabilizer (OIS) terminates this possibility by adjusting the images during the capturing. In this work, we propose a deep network that compensates the motions caused by the OIS, such that the gyroscopes can be used for image alignment on the OIS cameras. To achieve this, first, we record both videos and gyroscopes with an OIS camera as training data. Then, we convert gyroscope readings into motion fields. Second, we propose a Fundamental Mixtures motion model for rolling shutter cameras, where an array of rotations within a frame are extracted as the ground-truth guidance. Third, we train a convolutional neural network with gyroscope motions as input to compensate for the OIS motion. Once finished, the compensation network can be applied for other scenes, where the image alignment is purely based on gyroscopes with no need for images contents, delivering strong robustness. Experiments show that our results are comparable with that of non-OIS cameras, and outperform image-based alignment results with a relatively large margin. Code and dataset are available at https://github.com/lhaippp/DeepOIS