DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Model
This enables high-quality imaging in compact, lightweight cameras, with potential applications in other imaging systems.
The authors tackled the problem of poor image quality in flat lensless cameras by using a pre-trained diffusion model with a control network and learned transformation for reconstruction, achieving state-of-the-art results in quality and perceptuality.
The flat lensless camera design reduces the camera size and weight significantly. In this design, the camera lens is replaced by another optical element that interferes with the incoming light. The image is recovered from the raw sensor measurements using a reconstruction algorithm. Yet, the quality of the reconstructed images is not satisfactory. To mitigate this, we propose utilizing a pre-trained diffusion model with a control network and a learned separable transformation for reconstruction. This allows us to build a prototype flat camera with high-quality imaging, presenting state-of-the-art results in both terms of quality and perceptuality. We demonstrate its ability to leverage also textual descriptions of the captured scene to further enhance reconstruction. Our reconstruction method which leverages the strong capabilities of a pre-trained diffusion model can be used in other imaging systems for improved reconstruction results.