Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image
  Diffusion Models
Shweta Mahajan Tanzila Rahman Kwang Moo Yi Leonid Sigal
The quality of the prompts provided to text-to-image diffusion models
determines how faithful the generated content is to the user's intent, often
requiring `prompt engineering'. To harness visual concepts from target images
without prompt engineering, current approaches largely rely on embedding
inversion by optimizing and then mapping them to pseudo-tokens. However,
working with such high-dimensional vector representations is challenging
because they lack semantics and interpretability, and only allow simple vector
operations when using them. Instead, this work focuses on inverting the
diffusion model to obtain interpretable language prompts directly. The
challenge of doing this lies in the fact that the resulting optimization
problem is fundamentally discrete and the space of prompts is exponentially
large; this makes using standard optimization techniques, such as stochastic
gradient descent, difficult. To this end, we utilize a delayed projection
scheme to optimize for prompts representative of the vocabulary space in the
model. Further, we leverage the findings that different timesteps of the
diffusion process cater to different levels of detail in an image. The later,
noisy, timesteps of the forward diffusion process correspond to the semantic
information, and therefore, prompt inversion in this range provides tokens
representative of the image semantics. We show that our approach can identify
semantically interpretable and meaningful prompts for a target image which can
be used to synthesize diverse images with similar content. We further
illustrate the application of the optimized prompts in evolutionary image
generation and concept removal.