Manipulating Embeddings of Stable Diffusion Prompts
Niklas Deckers Julia Peters Martin Potthast
Generative text-to-image models such as Stable Diffusion allow users to
generate images based on a textual description, the prompt. Changing the prompt
is still the primary means for the user to change a generated image as desired.
However, changing the image by reformulating the prompt remains a difficult
process of trial and error, which has led to the emergence of prompt
engineering as a new field of research. We propose and analyze methods to
change the embedding of a prompt directly instead of the prompt text. It allows
for more fine-grained and targeted control that takes into account user
intentions. Our approach treats the generative text-to-image model as a
continuous function and passes gradients between the image space and the prompt
embedding space. By addressing different user interaction problems, we can
apply this idea in three scenarios: (1) Optimization of a metric defined in
image space that could measure, for example, image style. (2) Assistance of
users in creative tasks by enabling them to navigate the image space along a
selection of directions of "near" prompt embeddings. (3) Changing the embedding
of the prompt to include information that the user has seen in a particular
seed but finds difficult to describe in the prompt. Our experiments demonstrate
the feasibility of the described methods.