Hybrid Transformer Based Feature Fusion for Self-Supervised Monocular Depth Estimation
This work addresses the need for practical depth estimation in 3D vision applications by improving accuracy without requiring paired stereo data, representing an incremental advancement over existing self-supervised methods.
The paper tackles the problem of self-supervised monocular depth estimation by addressing the limitation of ResNet-based encoders in capturing only local information, proposing a hybrid model that fuses local features from convolutional encoders with global contextual information from a Vision Transformer encoder. The result is state-of-the-art performance on most standard benchmarks, though no specific numerical gains are provided in the abstract.
With an unprecedented increase in the number of agents and systems that aim to navigate the real world using visual cues and the rising impetus for 3D Vision Models, the importance of depth estimation is hard to understate. While supervised methods remain the gold standard in the domain, the copious amount of paired stereo data required to train such models makes them impractical. Most State of the Art (SOTA) works in the self-supervised and unsupervised domain employ a ResNet-based encoder architecture to predict disparity maps from a given input image which are eventually used alongside a camera pose estimator to predict depth without direct supervision. The fully convolutional nature of ResNets makes them susceptible to capturing per-pixel local information only, which is suboptimal for depth prediction. Our key insight for doing away with this bottleneck is to use Vision Transformers, which employ self-attention to capture the global contextual information present in an input image. Our model fuses per-pixel local information learned using two fully convolutional depth encoders with global contextual information learned by a transformer encoder at different scales. It does so using a mask-guided multi-stream convolution in the feature space to achieve state-of-the-art performance on most standard benchmarks.