STDepthFormer: Predicting Spatio-temporal Depth from Video with a Self-supervised Transformer Model
This addresses the challenge of robust depth forecasting in autonomous driving or robotics, offering an incremental improvement by extending depth prediction to sequences with implicit motion forecasting.
The paper tackles the problem of predicting spatio-temporal depth sequences from video, rather than single frames, using a self-supervised transformer model with a novel objective function for consistency, achieving highly accurate results that outperform baselines on the KITTI benchmark.
In this paper, a self-supervised model that simultaneously predicts a sequence of future frames from video-input with a novel spatial-temporal attention (ST) network is proposed. The ST transformer network allows constraining both temporal consistency across future frames whilst constraining consistency across spatial objects in the image at different scales. This was not the case in prior works for depth prediction, which focused on predicting a single frame as output. The proposed model leverages prior scene knowledge such as object shape and texture similar to single-image depth inference methods, whilst also constraining the motion and geometry from a sequence of input images. Apart from the transformer architecture, one of the main contributions with respect to prior works lies in the objective function that enforces spatio-temporal consistency across a sequence of output frames rather than a single output frame. As will be shown, this results in more accurate and robust depth sequence forecasting. The model achieves highly accurate depth forecasting results that outperform existing baselines on the KITTI benchmark. Extensive ablation studies were performed to assess the effectiveness of the proposed techniques. One remarkable result of the proposed model is that it is implicitly capable of forecasting the motion of objects in the scene, rather than requiring complex models involving multi-object detection, segmentation and tracking.