Explaining CLIP through Co-Creative Drawings and Interaction
Varvara Guljajeva Mar Canet Solà Isaac Joseph Clarke
This paper analyses a visual archive of drawings produced by an interactive
robotic art installation where audience members narrated their dreams into a
system powered by CLIPdraw deep learning (DL) model that interpreted and
transformed their dreams into images. The resulting archive of prompt-image
pairs were examined and clustered based on concept representation accuracy. As
a result of the analysis, the paper proposes four groupings for describing and
explaining CLIP-generated results: clear concept, text-to-text as image,
indeterminacy and confusion, and lost in translation. This article offers a
glimpse into a collection of dreams interpreted, mediated and given form by
Artificial Intelligence (AI), showcasing oftentimes unexpected, visually
compelling or, indeed, the dream-like output of the system, with the emphasis
on processes and results of translations between languages, sign-systems and
various modules of the installation. In the end, the paper argues that proposed
clusters support better understanding of the neural model.