Deformable kernel networks for guided depth map upsampling
This addresses depth map upsampling for applications like 3D reconstruction and robotics, offering a novel method that improves accuracy and speed over existing approaches.
The paper tackles the problem of upsampling low-resolution depth maps using high-resolution color images by proposing a deformable kernel network (DKN) that learns sparse, spatially-variant kernels, achieving state-of-the-art performance with 3x3 kernels and a fast version (FDKN) that runs 17 times faster (0.01 seconds for a 640x480 image).
We address the problem of upsampling a low-resolution (LR) depth map using a registered high-resolution (HR) color image of the same scene. Previous methods based on convolutional neural networks (CNNs) combine nonlinear activations of spatially-invariant kernels to estimate structural details from LR depth and HR color images, and regress upsampling results directly from the networks. In this paper, we revisit the weighted averaging process that has been widely used to transfer structural details from hand-crafted visual features to LR depth maps. We instead learn explicitly sparse and spatially-variant kernels for this task. To this end, we propose a CNN architecture and its efficient implementation, called the deformable kernel network (DKN), that outputs sparse sets of neighbors and the corresponding weights adaptively for each pixel. We also propose a fast version of DKN (FDKN) that runs about 17 times faster (0.01 seconds for a HR image of size 640 x 480). Experimental results on standard benchmarks demonstrate the effectiveness of our approach. In particular, we show that the weighted averaging process with 3 x 3 kernels (i.e., aggregating 9 samples sparsely chosen) outperforms the state of the art by a significant margin.