MobiFuse: A High-Precision On-device Depth Perception System with Multi-Data Fusion
This addresses the need for accurate on-device depth sensing for applications like 3D reconstruction and segmentation, representing a strong specific gain in mobile computer vision.
The paper tackles the problem of high-precision depth perception on mobile devices by proposing MobiFuse, a system that fuses dual RGB and Time-of-Flight camera data, resulting in up to a 77.7% reduction in depth measurement errors.
We present MobiFuse, a high-precision depth perception system on mobile devices that combines dual RGB and Time-of-Flight (ToF) cameras. To achieve this, we leverage physical principles from various environmental factors to propose the Depth Error Indication (DEI) modality, characterizing the depth error of ToF and stereo-matching. Furthermore, we employ a progressive fusion strategy, merging geometric features from ToF and stereo depth maps with depth error features from the DEI modality to create precise depth maps. Additionally, we create a new ToF-Stereo depth dataset, RealToF, to train and validate our model. Our experiments demonstrate that MobiFuse excels over baselines by significantly reducing depth measurement errors by up to 77.7%. It also showcases strong generalization across diverse datasets and proves effectiveness in two downstream tasks: 3D reconstruction and 3D segmentation. The demo video of MobiFuse in real-life scenarios is available at the de-identified YouTube link(https://youtu.be/jy-Sp7T1LVs).