Who Handles Orientation? Investigating Invariance in Feature Matching
For practitioners in 3D computer vision, this work provides guidance on designing rotation-invariant matching pipelines, showing that descriptor-level invariance is sufficient and efficient.
The paper investigates where rotation invariance should be incorporated in sparse feature matching pipelines. It finds that learning rotation invariance in the descriptor yields similar performance to handling it in the matcher, but achieves invariance earlier, enabling faster matching. The resulting matchers achieve state-of-the-art performance on multi-modal, extreme, and satellite image matching benchmarks.
Finding matching keypoints between images is a core problem in 3D computer vision. However, modern matchers struggle with large in-plane rotations. A straightforward mitigation is to learn rotation invariance via data augmentation. However, it remains unclear at which stage rotation invariance should be incorporated. In this paper, we study this in the context of a modern sparse matching pipeline. We perform extensive experiments by training on a large collection of 3D vision datasets and evaluating on popular image matching benchmarks. Surprisingly, we find that incorporating rotation invariance already in the descriptor yields similar performance to handling it in the matcher. However, rotation invariance is achieved earlier in the matcher when it is learned in the descriptor, allowing for a faster rotation-invariant matcher. Further, we find that enforcing rotation invariance does not hurt upright performance when trained at scale. Finally, we study the emergence of rotation invariance through scale and find that increasing the training data size substantially improves generalization to rotated images. We release two matchers robust to in-plane rotations that achieve state-of-the-art performance on e.g. multi-modal (WxBS), extreme (HardMatch), and satellite image matching (SatAst). Code is available at https://github.com/davnords/loma.