Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines
Michael Toker Hadas Orgad Mor Ventura Dana Arad Yonatan Belinkov
Text-to-image diffusion models (T2I) use a latent representation of a text
prompt to guide the image generation process. However, the process by which the
encoder produces the text representation is unknown. We propose the Diffusion
Lens, a method for analyzing the text encoder of T2I models by generating
images from its intermediate representations. Using the Diffusion Lens, we
perform an extensive analysis of two recent T2I models. Exploring compound
prompts, we find that complex scenes describing multiple objects are composed
progressively and more slowly compared to simple scenes; Exploring knowledge
retrieval, we find that representation of uncommon concepts requires further
computation compared to common concepts, and that knowledge retrieval is
gradual across layers. Overall, our findings provide valuable insights into the
text encoder component in T2I pipelines.