PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy
This addresses the need for accurate and relightable 3D reconstructions in endoscopy, potentially improving diagnosis and simulation tools for colorectal cancer and other conditions, though it appears incremental as it builds on existing 3D Gaussian Splatting methods.
The paper tackles the problem of 3D reconstruction from endoscopic images, which suffers from artifacts due to constrained camera trajectories and view-dependent lighting, by presenting PR-ENDO, a framework that enhances 3D Gaussian Splatting with a physically based relightable model to separate light effects from tissue properties, achieving superior reconstruction quality on public and in-house datasets.
Endoluminal endoscopic procedures are essential for diagnosing colorectal cancer and other severe conditions in the digestive tract, urogenital system, and airways. 3D reconstruction and novel-view synthesis from endoscopic images are promising tools for enhancing diagnosis. Moreover, integrating physiological deformations and interaction with the endoscope enables the development of simulation tools from real video data. However, constrained camera trajectories and view-dependent lighting create artifacts, leading to inaccurate or overfitted reconstructions. We present PR-ENDO, a novel 3D reconstruction framework leveraging the unique property of endoscopic imaging, where a single light source is closely aligned with the camera. Our method separates light effects from tissue properties. PR-ENDO enhances 3D Gaussian Splatting with a physically based relightable model. We boost the traditional light transport formulation with a specialized MLP capturing complex light-related effects while ensuring reduced artifacts and better generalization across novel views. PR-ENDO achieves superior reconstruction quality compared to baseline methods on both public and in-house datasets. Unlike existing approaches, PR-ENDO enables tissue modifications while preserving a physically accurate response to light, making it closer to real-world clinical use.