Improving In-Context Learning in Diffusion Models with Visual
  Context-Modulated Prompts
Tianqi Chen Yongfei Liu Zhendong Wang Jianbo Yuan Quanzeng You Hongxia Yang Mingyuan Zhou
In light of the remarkable success of in-context learning in large language
models, its potential extension to the vision domain, particularly with visual
foundation models like Stable Diffusion, has sparked considerable interest.
Existing approaches in visual in-context learning frequently face hurdles such
as expensive pretraining, limiting frameworks, inadequate visual comprehension,
and limited adaptability to new tasks. In response to these challenges, we
introduce improved Prompt Diffusion (iPromptDiff) in this study. iPromptDiff
integrates an end-to-end trained vision encoder that converts visual context
into an embedding vector. This vector is subsequently used to modulate the
token embeddings of text prompts. We show that a diffusion-based vision
foundation model, when equipped with this visual context-modulated text
guidance and a standard ControlNet structure, exhibits versatility and
robustness across a variety of training tasks and excels in in-context learning
for novel vision tasks, such as normal-to-image or image-to-line
transformations. The effectiveness of these capabilities relies heavily on a
deep visual understanding, which is achieved through relevant visual
demonstrations processed by our proposed in-context learning architecture.