CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation
This addresses the challenge of generating 3D shapes from text for creative applications, but it is incremental as it builds on existing CLIP and shape generation techniques.
The paper tackles the problem of text-to-shape generation without paired data by proposing CLIP-Forge, a two-stage method that uses unlabelled shapes and a pre-trained CLIP model, achieving promising zero-shot results with the ability to generate multiple shapes per text.
Generating shapes using natural language can enable new ways of imagining and creating the things around us. While significant recent progress has been made in text-to-image generation, text-to-shape generation remains a challenging problem due to the unavailability of paired text and shape data at a large scale. We present a simple yet effective method for zero-shot text-to-shape generation that circumvents such data scarcity. Our proposed method, named CLIP-Forge, is based on a two-stage training process, which only depends on an unlabelled shape dataset and a pre-trained image-text network such as CLIP. Our method has the benefits of avoiding expensive inference time optimization, as well as the ability to generate multiple shapes for a given text. We not only demonstrate promising zero-shot generalization of the CLIP-Forge model qualitatively and quantitatively, but also provide extensive comparative evaluations to better understand its behavior.