Robust Depth Enhancement via Polarization Prompt Fusion Tuning
This work addresses depth enhancement for sensors in computer vision, but it is incremental as it builds on existing polarization-based methods with a novel tuning approach.
The paper tackles the problem of inaccurate depth measurements from sensors in challenging scenarios like transparent or reflective objects by proposing a framework that uses polarization imaging and a learning-based neural network, enhanced with a Polarization Prompt Fusion Tuning strategy, and shows favorable performance compared to existing baselines in experiments on a public dataset.
Existing depth sensors are imperfect and may provide inaccurate depth values in challenging scenarios, such as in the presence of transparent or reflective objects. In this work, we present a general framework that leverages polarization imaging to improve inaccurate depth measurements from various depth sensors. Previous polarization-based depth enhancement methods focus on utilizing pure physics-based formulas for a single sensor. In contrast, our method first adopts a learning-based strategy where a neural network is trained to estimate a dense and complete depth map from polarization data and a sensor depth map from different sensors. To further improve the performance, we propose a Polarization Prompt Fusion Tuning (PPFT) strategy to effectively utilize RGB-based models pre-trained on large-scale datasets, as the size of the polarization dataset is limited to train a strong model from scratch. We conducted extensive experiments on a public dataset, and the results demonstrate that the proposed method performs favorably compared to existing depth enhancement baselines. Code and demos are available at https://lastbasket.github.io/PPFT/.