Understanding and Optimizing Attention-Based Sparse Matching for Diverse Local Features
This addresses deployment challenges for transformer-based matching models in computer vision by creating detector-agnostic solutions.
The paper identified that detectors, not descriptors, are the primary performance bottleneck in attention-based sparse image matching models, and proposed a fine-tuning approach using diverse detector keypoints to create a universal model that achieves or exceeds specialized model accuracy in zero-shot novel detector matching.
We revisit the problem of training attention-based sparse image matching models for various local features. We first identify one critical design choice that has been previously overlooked, which significantly impacts the performance of the LightGlue model. We then investigate the role of detectors and descriptors within the transformer-based matching framework, finding that detectors, rather than descriptors, are often the primary cause for performance difference. Finally, we propose a novel approach to fine-tune existing image matching models using keypoints from a diverse set of detectors, resulting in a universal, detector-agnostic model. When deployed as a zero-shot matcher for novel detectors, the resulting model achieves or exceeds the accuracy of models specifically trained for those features. Our findings offer valuable insights for the deployment of transformer-based matching models and the future design of local features.