MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion
  Models
Nithin Gopalakrishnan Nair Jeya Maria Jose Valanarasu Vishal M Patel
Large diffusion-based Text-to-Image (T2I) models have shown impressive
generative powers for text-to-image generation as well as spatially conditioned
image generation. For most applications, we can train the model end-toend with
paired data to obtain photorealistic generation quality. However, to add an
additional task, one often needs to retrain the model from scratch using paired
data across all modalities to retain good generation performance. In this
paper, we tackle this issue and propose a novel strategy to scale a generative
model across new tasks with minimal compute. During our experiments, we
discovered that the variance maps of intermediate feature maps of diffusion
models capture the intensity of conditioning. Utilizing this prior information,
we propose MaxFusion, an efficient strategy to scale up text-to-image
generation models to accommodate new modality conditions. Specifically, we
combine aligned features of multiple models, hence bringing a compositional
effect. Our fusion strategy can be integrated into off-the-shelf models to
enhance their generative prowess.