Endoscopic Depth Measurement and Super-Spectral-Resolution Imaging
This addresses the need for intra-operative tissue shape and spectral information in surgery, though it is incremental as it integrates existing techniques like SfM and CNNs.
The researchers developed an endoscopic system that combines structured lighting with hyperspectral imaging to enable real-time, scaled 3D surface reconstruction and dense hyperspectral data recovery from sparse measurements, validated on animal and human tissues.
Intra-operative measurements of tissue shape and multi/ hyperspectral information have the potential to provide surgical guidance and decision making support. We report an optical probe based system to combine sparse hyperspectral measurements and spectrally-encoded structured lighting (SL) for surface measurements. The system provides informative signals for navigation with a surgical interface. By rapidly switching between SL and white light (WL) modes, SL information is combined with structure-from-motion (SfM) from white light images, based on SURF feature detection and Lucas-Kanade (LK) optical flow to provide quasi-dense surface shape reconstruction with known scale in real-time. Furthermore, "super-spectral-resolution" was realized, whereby the RGB images and sparse hyperspectral data were integrated to recover dense pixel-level hyperspectral stacks, by using convolutional neural networks to upscale the wavelength dimension. Validation and demonstration of this system is reported on ex vivo/in vivo animal/ human experiments.