Unsupervised Homography Estimation on Multimodal Image Pair via Alternating Optimization
This addresses the challenge of homography estimation for multimodal images, which is crucial for tasks like image stitching and fusion, but it is incremental as it builds on existing unsupervised approaches.
The paper tackles the problem of estimating homography for multimodal image pairs without ground-truth data, proposing an unsupervised framework called AltO that uses alternating optimization with Barlow Twins losses, and it outperforms other unsupervised methods while being compatible with various architectures.
Estimating the homography between two images is crucial for mid- or high-level vision tasks, such as image stitching and fusion. However, using supervised learning methods is often challenging or costly due to the difficulty of collecting ground-truth data. In response, unsupervised learning approaches have emerged. Most early methods, though, assume that the given image pairs are from the same camera or have minor lighting differences. Consequently, while these methods perform effectively under such conditions, they generally fail when input image pairs come from different domains, referred to as multimodal image pairs. To address these limitations, we propose AltO, an unsupervised learning framework for estimating homography in multimodal image pairs. Our method employs a two-phase alternating optimization framework, similar to Expectation-Maximization (EM), where one phase reduces the geometry gap and the other addresses the modality gap. To handle these gaps, we use Barlow Twins loss for the modality gap and propose an extended version, Geometry Barlow Twins, for the geometry gap. As a result, we demonstrate that our method, AltO, can be trained on multimodal datasets without any ground-truth data. It not only outperforms other unsupervised methods but is also compatible with various architectures of homography estimators. The source code can be found at:~\url{https://github.com/songsang7/AltO}