E-M3RF: An Equivariant Multimodal 3D Re-assembly Framework
This work addresses a fundamental geometric problem in computer vision and cultural heritage preservation, offering a novel solution for reassembling fractured objects with improved accuracy.
The paper tackles the problem of 3D reassembly when geometry alone is insufficient or ambiguous, such as for small, eroded, or symmetric fragments, by introducing E-M3RF, an equivariant multimodal framework that uses both geometric and color features, resulting in a 23.1% reduction in rotation error, 13.2% reduction in translation error, and 18.4% decrease in Chamfer Distance on the RePAIR dataset compared to competing methods.
3D reassembly is a fundamental geometric problem, and in recent years it has increasingly been challenged by deep learning methods rather than classical optimization. While learning approaches have shown promising results, most still rely primarily on geometric features to assemble a whole from its parts. As a result, methods struggle when geometry alone is insufficient or ambiguous, for example, for small, eroded, or symmetric fragments. Additionally, solutions do not impose physical constraints that explicitly prevent overlapping assemblies. To address these limitations, we introduce E-M3RF, an equivariant multimodal 3D reassembly framework that takes as input the point clouds, containing both point positions and colors of fractured fragments, and predicts the transformations required to reassemble them using SE(3) flow matching. Each fragment is represented by both geometric and color features: i) 3D point positions are encoded as rotationconsistent geometric features using a rotation-equivariant encoder, ii) the colors at each 3D point are encoded with a transformer. The two feature sets are then combined to form a multimodal representation. We experimented on four datasets: two synthetic datasets, Breaking Bad and Fantastic Breaks, and two real-world cultural heritage datasets, RePAIR and Presious, demonstrating that E-M3RF on the RePAIR dataset reduces rotation error by 23.1% and translation error by 13.2%, while Chamfer Distance decreases by 18.4% compared to competing methods.