ReassembleNet: Learnable Keypoints and Diffusion for 2D Fresco Reconstruction
This work addresses the reassembly challenge for domains like archaeology, offering a method that handles complex shapes and erosion, though it appears incremental with specific gains.
The paper tackled the problem of reassembling 2D frescoes by addressing scalability, multimodality, and real-world applicability limitations in deep learning methods, resulting in improvements of 57% and 87% for RMSE rotation and translation errors, respectively.
The task of reassembly is a significant challenge across multiple domains, including archaeology, genomics, and molecular docking, requiring the precise placement and orientation of elements to reconstruct an original structure. In this work, we address key limitations in state-of-the-art Deep Learning methods for reassembly, namely i) scalability; ii) multimodality; and iii) real-world applicability: beyond square or simple geometric shapes, realistic and complex erosion, or other real-world problems. We propose ReassembleNet, a method that reduces complexity by representing each input piece as a set of contour keypoints and learning to select the most informative ones by Graph Neural Networks pooling inspired techniques. ReassembleNet effectively lowers computational complexity while enabling the integration of features from multiple modalities, including both geometric and texture data. Further enhanced through pretraining on a semi-synthetic dataset. We then apply diffusion-based pose estimation to recover the original structure. We improve on prior methods by 57% and 87% for RMSE Rotation and Translation, respectively.