Composition and Deformance: Measuring Imageability with a Text-to-Image
  Model
Si Wu David A. Smith
Although psycholinguists and psychologists have long studied the tendency of
linguistic strings to evoke mental images in hearers or readers, most
computational studies have applied this concept of imageability only to
isolated words. Using recent developments in text-to-image generation models,
such as DALLE mini, we propose computational methods that use generated images
to measure the imageability of both single English words and connected text. We
sample text prompts for image generation from three corpora: human-generated
image captions, news article sentences, and poem lines. We subject these
prompts to different deformances to examine the model's ability to detect
changes in imageability caused by compositional change. We find high
correlation between the proposed computational measures of imageability and
human judgments of individual words. We also find the proposed measures more
consistently respond to changes in compositionality than baseline approaches.
We discuss possible effects of model training and implications for the study of
compositionality in text-to-image models.