Scene Graph Conditioning in Latent Diffusion
Frank Fundel
Diffusion models excel in image generation but lack detailed semantic control
using text prompts. Additional techniques have been developed to address this
limitation. However, conditioning diffusion models solely on text-based
descriptions is challenging due to ambiguity and lack of structure. In
contrast, scene graphs offer a more precise representation of image content,
making them superior for fine-grained control and accurate synthesis in image
generation models. The amount of image and scene-graph data is sparse, which
makes fine-tuning large diffusion models challenging. We propose multiple
approaches to tackle this problem using ControlNet and Gated Self-Attention. We
were able to show that using out proposed methods it is possible to generate
images from scene graphs with much higher quality, outperforming previous
methods. Our source code is publicly available on
https://github.com/FrankFundel/SGCond