Improving Explicit Spatial Relationships in Text-to-Image Generation
  through an Automatically Derived Dataset
Ander Salaberria Gorka Azkune Oier Lopez de Lacalle Aitor Soroa Eneko Agirre Frank Keller
Existing work has observed that current text-to-image systems do not
accurately reflect explicit spatial relations between objects such as 'left of'
or 'below'. We hypothesize that this is because explicit spatial relations
rarely appear in the image captions used to train these models. We propose an
automatic method that, given existing images, generates synthetic captions that
contain 14 explicit spatial relations. We introduce the Spatial Relation for
Generation (SR4G) dataset, which contains 9.9 millions image-caption pairs for
training, and more than 60 thousand captions for evaluation. In order to test
generalization we also provide an 'unseen' split, where the set of objects in
the train and test captions are disjoint. SR4G is the first dataset that can be
used to spatially fine-tune text-to-image systems. We show that fine-tuning two
different Stable Diffusion models (denoted as SD$_{SR4G}$) yields up to 9
points improvements in the VISOR metric. The improvement holds in the 'unseen'
split, showing that SD$_{SR4G}$ is able to generalize to unseen objects.
SD$_{SR4G}$ improves the state-of-the-art with fewer parameters, and avoids
complex architectures. Our analysis shows that improvement is consistent for
all relations. The dataset and the code will be publicly available.