Augment the Pairs: Semantics-Preserving Image-Caption Pair Augmentation
  for Grounding-Based Vision and Language Models
Jingru Yi Burak Uzkent Oana Ignat Zili Li Amanmeet Garg Xiang Yu Linda Liu
Grounding-based vision and language models have been successfully applied to
low-level vision tasks, aiming to precisely locate objects referred in
captions. The effectiveness of grounding representation learning heavily relies
on the scale of the training dataset. Despite being a useful data enrichment
strategy, data augmentation has received minimal attention in existing vision
and language tasks as augmentation for image-caption pairs is non-trivial. In
this study, we propose a robust phrase grounding model trained with
text-conditioned and text-unconditioned data augmentations. Specifically, we
apply text-conditioned color jittering and horizontal flipping to ensure
semantic consistency between images and captions. To guarantee image-caption
correspondence in the training samples, we modify the captions according to
pre-defined keywords when applying horizontal flipping. Additionally, inspired
by recent masked signal reconstruction, we propose to use pixel-level masking
as a novel form of data augmentation. While we demonstrate our data
augmentation method with MDETR framework, the proposed approach is applicable
to common grounding-based vision and language tasks with other frameworks.
Finally, we show that image encoder pretrained on large-scale image and
language datasets (such as CLIP) can further improve the results. Through
extensive experiments on three commonly applied datasets: Flickr30k, referring
expressions and GQA, our method demonstrates advanced performance over the
state-of-the-arts with various metrics. Code can be found in
https://github.com/amzn/augment-the-pairs-wacv2024.