Deepzzle: Solving Visual Jigsaw Puzzles with Deep Learning andShortest Path Optimization
This work addresses the challenge of reconstructing images from fragmented pieces, particularly for cultural heritage applications, but is incremental in its approach.
The paper tackles the image reassembly problem with widely spaced fragments, emulating archaeological erosion, by using a two-step method involving a neural network for position prediction and graph optimization, and introduces a new metric for evaluation.
We tackle the image reassembly problem with wide space between the fragments, in such a way that the patterns and colors continuity is mostly unusable. The spacing emulates the erosion of which the archaeological fragments suffer. We crop-square the fragments borders to compel our algorithm to learn from the content of the fragments. We also complicate the image reassembly by removing fragments and adding pieces from other sources. We use a two-step method to obtain the reassemblies: 1) a neural network predicts the positions of the fragments despite the gaps between them; 2) a graph that leads to the best reassemblies is made from these predictions. In this paper, we notably investigate the effect of branch-cut in the graph of reassemblies. We also provide a comparison with the literature, solve complex images reassemblies, explore at length the dataset, and propose a new metric that suits its specificities. Keywords: image reassembly, jigsaw puzzle, deep learning, graph, branch-cut, cultural heritage