CLIPDrawX: Primitive-based Explanations for Text Guided Sketch Synthesis
Nityanand Mathur Shyam Marjit Abhra Chaudhuri Anjan Dutta
With the goal of understanding the visual concepts that CLIP associates with
text prompts, we show that the latent space of CLIP can be visualized solely in
terms of linear transformations on simple geometric primitives like circles and
straight lines. Although existing approaches achieve this by
sketch-synthesis-through-optimization, they do so on the space of B\'ezier
curves, which exhibit a wastefully large set of structures that they can evolve
into, as most of them are non-essential for generating meaningful sketches. We
present CLIPDrawX, an algorithm that provides significantly better
visualizations for CLIP text embeddings, using only simple primitive shapes
like straight lines and circles. This constrains the set of possible outputs to
linear transformations on these primitives, thereby exhibiting an inherently
simpler mathematical form. The synthesis process of CLIPDrawX can be tracked
end-to-end, with each visual concept being explained exclusively in terms of
primitives. Implementation will be released upon acceptance. Project Page:
$\href{https://clipdrawx.github.io/}{\text{https://clipdrawx.github.io/}}$.