Image to Icosahedral Projection for $\mathrm{SO}(3)$ Object Reasoning from Single-View Images
This addresses the challenge of imposing equivariance in models for tasks like pose estimation from 2D images, which is incremental as it builds on existing equivariant models but extends them to image input.
The paper tackled the problem of 3D object reasoning from single-view 2D images by proposing a novel architecture that projects images onto an icosahedron to achieve approximate SO(3) equivariance, and it outperformed baselines on object pose estimation and shape classification tasks.
Reasoning about 3D objects based on 2D images is challenging due to variations in appearance caused by viewing the object from different orientations. Tasks such as object classification are invariant to 3D rotations and other such as pose estimation are equivariant. However, imposing equivariance as a model constraint is typically not possible with 2D image input because we do not have an a priori model of how the image changes under out-of-plane object rotations. The only $\mathrm{SO}(3)$-equivariant models that currently exist require point cloud or voxel input rather than 2D images. In this paper, we propose a novel architecture based on icosahedral group convolutions that reasons in $\mathrm{SO(3)}$ by learning a projection of the input image onto an icosahedron. The resulting model is approximately equivariant to rotation in $\mathrm{SO}(3)$. We apply this model to object pose estimation and shape classification tasks and find that it outperforms reasonable baselines. Project website: \url{https://dmklee.github.io/image2icosahedral}