CLIPDraw++: Text-to-Sketch Synthesis with Simple Primitives
This work provides a more interpretable and efficient method for understanding CLIP's visual associations, though it is incremental as it builds on prior sketch-synthesis techniques.
The authors tackled the problem of visualizing CLIP's text embeddings by constraining sketch synthesis to simple geometric primitives like lines and circles, resulting in significantly better visualizations compared to methods using higher-order Bézier curves.
With the goal of understanding the visual concepts that CLIP associates with text prompts, we show that the latent space of CLIP can be visualized solely in terms of linear transformations on simple geometric primitives like straight lines and circles. Although existing approaches achieve this by sketch-synthesis-through-optimization, they do so on the space of higher order Bézier curves, which exhibit a wastefully large set of structures that they can evolve into, as most of them are non-essential for generating meaningful sketches. We present CLIPDraw++, an algorithm that provides significantly better visualizations for CLIP text embeddings, using only simple primitive shapes like straight lines and circles. This constrains the set of possible outputs to linear transformations on these primitives, thereby exhibiting an inherently simpler mathematical form. The synthesis process of CLIPDraw++ can be tracked end-to-end, with each visual concept being expressed exclusively in terms of primitives. Project Page: https://clipdrawx.github.io/.