Structure from Collision
This addresses a new task for 3D reconstruction in computer vision, enabling estimation of hidden internal structures, which is incremental as it builds on neural 3D representations.
The paper tackles the problem of estimating invisible internal structures of objects, which is difficult with existing neural 3D representations like NeRF and 3DGS, by introducing Structure from Collision (SfC) that uses appearance changes during collisions. The proposed SfC-NeRF model, with volume annealing to avoid local optima, was tested on 115 objects with diverse cavity shapes and material properties, demonstrating its effectiveness.
Recent advancements in neural 3D representations, such as neural radiance fields (NeRF) and 3D Gaussian splatting (3DGS), have enabled the accurate estimation of 3D structures from multiview images. However, this capability is limited to estimating the visible external structure, and identifying the invisible internal structure hidden behind the surface is difficult. To overcome this limitation, we address a new task called Structure from Collision (SfC), which aims to estimate the structure (including the invisible internal structure) of an object from appearance changes during collision. To solve this problem, we propose a novel model called SfC-NeRF that optimizes the invisible internal structure of an object through a video sequence under physical, appearance (i.e., visible external structure)-preserving, and keyframe constraints. In particular, to avoid falling into undesirable local optima owing to its ill-posed nature, we propose volume annealing; that is, searching for global optima by repeatedly reducing and expanding the volume. Extensive experiments on 115 objects involving diverse structures (i.e., various cavity shapes, locations, and sizes) and material properties revealed the properties of SfC and demonstrated the effectiveness of the proposed SfC-NeRF.