GANESH: Generalizable NeRF for Lensless Imaging
This work addresses the challenge of developing compact cameras for lensless imaging by enabling generalizable 3D reconstruction, though it is incremental as it builds on existing lensless imaging techniques.
The paper tackles the problem of 3D reconstruction from lensless images, which typically produce multiplexed outputs, by introducing GANESH, a framework that enables simultaneous refinement and novel view synthesis without scene-specific training, outperforming current methods in reconstruction accuracy and refinement quality.
Lensless imaging offers a significant opportunity to develop ultra-compact cameras by removing the conventional bulky lens system. However, without a focusing element, the sensor's output is no longer a direct image but a complex multiplexed scene representation. Traditional methods have attempted to address this challenge by employing learnable inversions and refinement models, but these methods are primarily designed for 2D reconstruction and do not generalize well to 3D reconstruction. We introduce GANESH, a novel framework designed to enable simultaneous refinement and novel view synthesis from multi-view lensless images. Unlike existing methods that require scene-specific training, our approach supports on-the-fly inference without retraining on each scene. Moreover, our framework allows us to tune our model to specific scenes, enhancing the rendering and refinement quality. To facilitate research in this area, we also present the first multi-view lensless dataset, LenslessScenes. Extensive experiments demonstrate that our method outperforms current approaches in reconstruction accuracy and refinement quality. Code and video results are available at https://rakesh-123-cryp.github.io/Rakesh.github.io/