Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration
This work provides a novel approach for improving depth estimation and image restoration from dual-pixel data, which is beneficial for computer vision systems utilizing such hardware, particularly in scenarios with significant defocus blur.
This paper addresses the challenge of estimating depth and restoring images from dual-pixel (DP) data, where defocus blur affects traditional stereo matching. The authors propose a mathematical DP model that leverages blur for depth estimation and introduce DDDNet, an end-to-end network that jointly estimates depth and restores images. Their method achieves competitive performance against state-of-the-art approaches on both synthetic and real datasets.
The dual-pixel (DP) hardware works by splitting each pixel in half and creating an image pair in a single snapshot. Several works estimate depth/inverse depth by treating the DP pair as a stereo pair. However, dual-pixel disparity only occurs in image regions with the defocus blur. The heavy defocus blur in DP pairs affects the performance of matching-based depth estimation approaches. Instead of removing the blur effect blindly, we study the formation of the DP pair which links the blur and the depth information. In this paper, we propose a mathematical DP model which can benefit depth estimation by the blur. These explorations motivate us to propose an end-to-end DDDNet (DP-based Depth and Deblur Network) to jointly estimate the depth and restore the image. Moreover, we define a reblur loss, which reflects the relationship of the DP image formation process with depth information, to regularise our depth estimate in training. To meet the requirement of a large amount of data for learning, we propose the first DP image simulator which allows us to create datasets with DP pairs from any existing RGBD dataset. As a side contribution, we collect a real dataset for further research. Extensive experimental evaluation on both synthetic and real datasets shows that our approach achieves competitive performance compared to state-of-the-art approaches.