An Image is Worth Multiple Words: Multi-attribute Inversion for
  Constrained Text-to-Image Synthesis
Aishwarya Agarwal Srikrishna Karanam Tripti Shukla Balaji Vasan Srinivasan
We consider the problem of constraining diffusion model outputs with a
user-supplied reference image. Our key objective is to extract multiple
attributes (e.g., color, object, layout, style) from this single reference
image, and then generate new samples with them. One line of existing work
proposes to invert the reference images into a single textual conditioning
vector, enabling generation of new samples with this learned token. These
methods, however, do not learn multiple tokens that are necessary to condition
model outputs on the multiple attributes noted above. Another line of
techniques expand the inversion space to learn multiple embeddings but they do
this only along the layer dimension (e.g., one per layer of the DDPM model) or
the timestep dimension (one for a set of timesteps in the denoising process),
leading to suboptimal attribute disentanglement. To address the aforementioned
gaps, the first contribution of this paper is an extensive analysis to
determine which attributes are captured in which dimension of the denoising
process. As noted above, we consider both the time-step dimension (in reverse
denoising) as well as the DDPM model layer dimension. We observe that often a
subset of these attributes are captured in the same set of model layers and/or
across same denoising timesteps. For instance, color and style are captured
across same U-Net layers, whereas layout and color are captured across same
timestep stages. Consequently, an inversion process that is designed only for
the time-step dimension or the layer dimension is insufficient to disentangle
all attributes. This leads to our second contribution where we design a new
multi-attribute inversion algorithm, MATTE, with associated
disentanglement-enhancing regularization losses, that operates across both
dimensions and explicitly leads to four disentangled tokens (color, style,
layout, and object).