Grounding Spatial Relations in Text-Only Language Models
Gorka Azkune Ander Salaberria Eneko Agirre
This paper shows that text-only Language Models (LM) can learn to ground
spatial relations like "left of" or "below" if they are provided with explicit
location information of objects and they are properly trained to leverage those
locations. We perform experiments on a verbalized version of the Visual Spatial
Reasoning (VSR) dataset, where images are coupled with textual statements which
contain real or fake spatial relations between two objects of the image. We
verbalize the images using an off-the-shelf object detector, adding location
tokens to every object label to represent their bounding boxes in textual form.
Given the small size of VSR, we do not observe any improvement when using
locations, but pretraining the LM over a synthetic dataset automatically
derived by us improves results significantly when using location tokens. We
thus show that locations allow LMs to ground spatial relations, with our
text-only LMs outperforming Vision-and-Language Models and setting the new
state-of-the-art for the VSR dataset. Our analysis show that our text-only LMs
can generalize beyond the relations seen in the synthetic dataset to some
extent, learning also more useful information than that encoded in the spatial
rules we used to create the synthetic dataset itself.