UFM: Unified Feature Matching Pre-training with Multi-Modal Image Assistants
This addresses feature matching problems for multimodal computer vision applications, but it appears incremental as it builds on existing pre-training and transformer methods.
The paper tackles the challenge of image feature matching across multimodal images by introducing a Unified Feature Matching pre-trained model (UFM) with Multimodal Image Assistant transformers, achieving strong generalization and performance in various tasks.
Image feature matching, a foundational task in computer vision, remains challenging for multimodal image applications, often necessitating intricate training on specific datasets. In this paper, we introduce a Unified Feature Matching pre-trained model (UFM) designed to address feature matching challenges across a wide spectrum of modal images. We present Multimodal Image Assistant (MIA) transformers, finely tunable structures adept at handling diverse feature matching problems. UFM exhibits versatility in addressing both feature matching tasks within the same modal and those across different modals. Additionally, we propose a data augmentation algorithm and a staged pre-training strategy to effectively tackle challenges arising from sparse data in specific modals and imbalanced modal datasets. Experimental results demonstrate that UFM excels in generalization and performance across various feature matching tasks. The code will be released at:https://github.com/LiaoYun0x0/UFM.