WT-MVSNet: Window-based Transformers for Multi-view Stereo
This work addresses multi-view stereo reconstruction for computer vision applications, presenting an incremental improvement by integrating transformers with epipolar constraints and a novel loss function.
The authors tackled the problem of multi-view stereo by proposing a window-based transformer approach for local feature matching and global feature aggregation, achieving state-of-the-art performance with a first-place ranking on the Tanks and Temples benchmark.
Recently, Transformers were shown to enhance the performance of multi-view stereo by enabling long-range feature interaction. In this work, we propose Window-based Transformers (WT) for local feature matching and global feature aggregation in multi-view stereo. We introduce a Window-based Epipolar Transformer (WET) which reduces matching redundancy by using epipolar constraints. Since point-to-line matching is sensitive to erroneous camera pose and calibration, we match windows near the epipolar lines. A second Shifted WT is employed for aggregating global information within cost volume. We present a novel Cost Transformer (CT) to replace 3D convolutions for cost volume regularization. In order to better constrain the estimated depth maps from multiple views, we further design a novel geometric consistency loss (Geo Loss) which punishes unreliable areas where multi-view consistency is not satisfied. Our WT multi-view stereo method (WT-MVSNet) achieves state-of-the-art performance across multiple datasets and ranks $1^{st}$ on Tanks and Temples benchmark.