SYM3D: Learning Symmetric Triplanes for Better 3D-Awareness of GANs
This addresses the scarcity of calibrated multi-view datasets for 3D GANs, enabling better 3D generation from abundant single-view images, though it is incremental in improving existing methods.
The paper tackled the problem of 3D-aware GANs requiring multi-view images with camera annotations by proposing SYM3D, which leverages reflectional symmetry and a view-aware spatial attention mechanism to generate high-quality 3D assets from single-view images, demonstrating superior performance on synthetic and real-world datasets.
Despite the growing success of 3D-aware GANs, which can be trained on 2D images to generate high-quality 3D assets, they still rely on multi-view images with camera annotations to synthesize sufficient details from all viewing directions. However, the scarce availability of calibrated multi-view image datasets, especially in comparison to single-view images, has limited the potential of 3D GANs. Moreover, while bypassing camera pose annotations with a camera distribution constraint reduces dependence on exact camera parameters, it still struggles to generate a consistent orientation of 3D assets. To this end, we propose SYM3D, a novel 3D-aware GAN designed to leverage the prevalent reflectional symmetry structure found in natural and man-made objects, alongside a proposed view-aware spatial attention mechanism in learning the 3D representation. We evaluate SYM3D on both synthetic (ShapeNet Chairs, Cars, and Airplanes) and real-world datasets (ABO-Chair), demonstrating its superior performance in capturing detailed geometry and texture, even when trained on only single-view images. Finally, we demonstrate the effectiveness of incorporating symmetry regularization in helping reduce artifacts in the modeling of 3D assets in the text-to-3D task. Project is at \url{https://jingyang2017.github.io/sym3d.github.io/}