Transformers in Unsupervised Structure-from-Motion
This work addresses 3D scene understanding for robotics and driver assistance systems, but it is incremental as it adapts existing transformers to a known task.
The paper tackles monocular structure-from-motion for 3D scene understanding by proposing a transformer-based method that predicts depth, ego-motion, and camera parameters simultaneously, achieving comparable performance to CNN-based methods on KITTI and DDAD datasets while being more robust to natural corruptions and attacks.
Transformers have revolutionized deep learning based computer vision with improved performance as well as robustness to natural corruptions and adversarial attacks. Transformers are used predominantly for 2D vision tasks, including image classification, semantic segmentation, and object detection. However, robots and advanced driver assistance systems also require 3D scene understanding for decision making by extracting structure-from-motion (SfM). We propose a robust transformer-based monocular SfM method that learns to predict monocular pixel-wise depth, ego vehicle's translation and rotation, as well as camera's focal length and principal point, simultaneously. With experiments on KITTI and DDAD datasets, we demonstrate how to adapt different vision transformers and compare them against contemporary CNN-based methods. Our study shows that transformer-based architecture, though lower in run-time efficiency, achieves comparable performance while being more robust against natural corruptions, as well as untargeted and targeted attacks.