ViPE: Visualise Pretty-much Everything
Hassan Shahmohammadi Adhiraj Ghosh Hendrik P. A. Lensch
Figurative and non-literal expressions are profoundly integrated in human
communication. Visualising such expressions allow us to convey our creative
thoughts, and evoke nuanced emotions. Recent text-to-image models like Stable
Diffusion, on the other hand, struggle to depict non-literal expressions.
Recent works primarily deal with this issue by compiling humanly annotated
datasets on a small scale, which not only demands specialised expertise but
also proves highly inefficient. To address this issue, we introduce ViPE:
Visualise Pretty-much Everything. ViPE offers a series of lightweight and
robust language models that have been trained on a large-scale set of lyrics
with noisy visual descriptions that represent their implicit meaning. The
synthetic visual descriptions are generated by GPT3.5 relying on neither human
annotations nor images. ViPE effectively expresses any arbitrary piece of text
into a visualisable description, enabling meaningful and high-quality image
generation. We provide compelling evidence that ViPE is more robust than GPT3.5
in synthesising visual elaborations. ViPE also exhibits an understanding of
figurative expressions comparable to human experts, providing a powerful and
open-source backbone to many downstream applications such as music video and
caption generation.