NeuralMarker: A Framework for Learning General Marker Correspondence
This addresses the problem of robust marker-image alignment for applications like AR and video editing, representing a novel method for a known bottleneck.
The paper tackles the problem of estimating correspondences from general markers (e.g., movie posters) to images, proposing NeuralMarker, a framework that trains a neural network for dense marker correspondence under challenging conditions like deformation and lighting, and shows it significantly outperforms previous methods, enabling applications in AR and video editing.
We tackle the problem of estimating correspondences from a general marker, such as a movie poster, to an image that captures such a marker. Conventionally, this problem is addressed by fitting a homography model based on sparse feature matching. However, they are only able to handle plane-like markers and the sparse features do not sufficiently utilize appearance information. In this paper, we propose a novel framework NeuralMarker, training a neural network estimating dense marker correspondences under various challenging conditions, such as marker deformation, harsh lighting, etc. Besides, we also propose a novel marker correspondence evaluation method circumstancing annotations on real marker-image pairs and create a new benchmark. We show that NeuralMarker significantly outperforms previous methods and enables new interesting applications, including Augmented Reality (AR) and video editing.