Estimating Extreme 3D Image Rotation with Transformer Cross-Attention
This work addresses a key challenge in computer vision for applications like image registration and 3D reconstruction, though it appears incremental as it builds on existing CNN and Transformer methods.
The paper tackles the problem of estimating extreme 3D image rotation between image pairs with limited or non-overlapping views, and proposes a cross-attention-based method that outperforms state-of-the-art schemes on common datasets, establishing new accuracy benchmarks.
The estimation of large and extreme image rotation plays a key role in multiple computer vision domains, where the rotated images are related by a limited or a non-overlapping field of view. Contemporary approaches apply convolutional neural networks to compute a 4D correlation volume to estimate the relative rotation between image pairs. In this work, we propose a cross-attention-based approach that utilizes CNN feature maps and a Transformer-Encoder, to compute the cross-attention between the activation maps of the image pairs, which is shown to be an improved equivalent of the 4D correlation volume, used in previous works. In the suggested approach, higher attention scores are associated with image regions that encode visual cues of rotation. Our approach is end-to-end trainable and optimizes a simple regression loss. It is experimentally shown to outperform contemporary state-of-the-art schemes when applied to commonly used image rotation datasets and benchmarks, and establishes a new state-of-the-art accuracy on these datasets. We make our code publicly available.