Selectively Informative Description can Reduce Undesired Embedding
  Entanglements in Text-to-Image Personalization
Jimyeong Kim Jungwon Park Wonjong Rhee
In text-to-image personalization, a timely and crucial challenge is the
tendency of generated images overfitting to the biases present in the reference
images. We initiate our study with a comprehensive categorization of the biases
into background, nearby-object, tied-object, substance (in style
re-contextualization), and pose biases. These biases manifest in the generated
images due to their entanglement into the subject embedding. This undesired
embedding entanglement not only results in the reflection of biases from the
reference images into the generated images but also notably diminishes the
alignment of the generated images with the given generation prompt. To address
this challenge, we propose SID~(Selectively Informative Description), a text
description strategy that deviates from the prevalent approach of only
characterizing the subject's class identification. SID is generated utilizing
multimodal GPT-4 and can be seamlessly integrated into optimization-based
models. We present comprehensive experimental results along with analyses of
cross-attention maps, subject-alignment, non-subject-disentanglement, and
text-alignment.