Probing the Role of Positional Information in Vision-Language Models
Philipp J. Rösch Jindřich Libovický
In most Vision-Language models (VL), the understanding of the image structure
is enabled by injecting the position information (PI) about objects in the
image. In our case study of LXMERT, a state-of-the-art VL model, we probe the
use of the PI in the representation and study its effect on Visual Question
Answering. We show that the model is not capable of leveraging the PI for the
image-text matching task on a challenge set where only position differs. Yet,
our experiments with probing confirm that the PI is indeed present in the
representation. We introduce two strategies to tackle this: (i) Positional
Information Pre-training and (ii) Contrastive Learning on PI using
Cross-Modality Matching. Doing so, the model can correctly classify if images
with detailed PI statements match. Additionally to the 2D information from
bounding boxes, we introduce the object's depth as new feature for a better
object localization in the space. Even though we were able to improve the model
properties as defined by our probes, it only has a negligible effect on the
downstream performance. Our results thus highlight an important issue of
multimodal modeling: the mere presence of information detectable by a probing
classifier is not a guarantee that the information is available in a
cross-modal setup.