HoloGarment: 360° Novel View Synthesis of In-the-Wild Garments
This addresses the problem of realistic 3D garment visualization for applications like virtual try-on or fashion, though it appears incremental as it builds on prior NVS methods with a new training paradigm.
The paper tackles novel view synthesis of in-the-wild garments, which is challenging due to occlusions, poses, and deformations, by proposing HoloGarment, a method that uses 1-3 images or video to generate 360° views in a canonical pose, achieving state-of-the-art performance with robust handling of real-world artifacts like wrinkling and occlusion.
Novel view synthesis (NVS) of in-the-wild garments is a challenging task due significant occlusions, complex human poses, and cloth deformations. Prior methods rely on synthetic 3D training data consisting of mostly unoccluded and static objects, leading to poor generalization on real-world clothing. In this paper, we propose HoloGarment (Hologram-Garment), a method that takes 1-3 images or a continuous video of a person wearing a garment and generates 360° novel views of the garment in a canonical pose. Our key insight is to bridge the domain gap between real and synthetic data with a novel implicit training paradigm leveraging a combination of large-scale real video data and small-scale synthetic 3D data to optimize a shared garment embedding space. During inference, the shared embedding space further enables dynamic video-to-360° NVS through the construction of a garment "atlas" representation by finetuning a garment embedding on a specific real-world video. The atlas captures garment-specific geometry and texture across all viewpoints, independent of body pose or motion. Extensive experiments show that HoloGarment achieves state-of-the-art performance on NVS of in-the-wild garments from images and videos. Notably, our method robustly handles challenging real-world artifacts -- such as wrinkling, pose variation, and occlusion -- while maintaining photorealism, view consistency, fine texture details, and accurate geometry. Visit our project page for additional results: https://johannakarras.github.io/HoloGarment