PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging
This work addresses the challenge of producing photorealistic images in lensless imaging, which is crucial for applications in compact and low-cost camera systems, though it is incremental as it builds on existing methods with a hybrid approach.
The paper tackles the problem of reconstructing high-quality images from lensless cameras, which suffer from inaccurate models and insufficient priors, by introducing a two-stage approach that combines spatially varying deconvolution for low-frequency consistency with a diffusion model for high-frequency photorealism, achieving superior balance between fidelity and visual quality on PhlatCam and DiffuserCam systems.
Lensless cameras offer significant advantages in size, weight, and cost compared to traditional lens-based systems. Without a focusing lens, lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, current algorithms struggle with inaccurate forward imaging models and insufficient priors to reconstruct high-quality images. To overcome these limitations, we introduce a novel two-stage approach for consistent and photorealistic lensless image reconstruction. The first stage of our approach ensures data consistency by focusing on accurately reconstructing the low-frequency content with a spatially varying deconvolution method that adjusts to changes in the Point Spread Function (PSF) across the camera's field of view. The second stage enhances photorealism by incorporating a generative prior from pre-trained diffusion models. By conditioning on the low-frequency content retrieved in the first stage, the diffusion model effectively reconstructs the high-frequency details that are typically lost in the lensless imaging process, while also maintaining image fidelity. Our method achieves a superior balance between data fidelity and visual quality compared to existing methods, as demonstrated with two popular lensless systems, PhlatCam and DiffuserCam. Project website: https://phocolens.github.io/.