GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language
  Pre-training and Open-Vocabulary Object Detection
Haozhan Shen Tiancheng Zhao Mingwei Zhu Jianwei Yin
Visual grounding, a crucial vision-language task involving the understanding
of the visual context based on the query expression, necessitates the model to
capture the interactions between objects, as well as various spatial and
attribute information. However, the annotation data of visual grounding task is
limited due to its time-consuming and labor-intensive annotation process,
resulting in the trained models being constrained from generalizing its
capability to a broader domain. To address this challenge, we propose
GroundVLP, a simple yet effective zero-shot method that harnesses visual
grounding ability from the existing models trained from image-text pairs and
pure object detection data, both of which are more conveniently obtainable and
offer a broader domain compared to visual grounding annotation data. GroundVLP
proposes a fusion mechanism that combines the heatmap from GradCAM and the
object proposals of open-vocabulary detectors. We demonstrate that the proposed
method significantly outperforms other zero-shot methods on RefCOCO/+/g
datasets, surpassing prior zero-shot state-of-the-art by approximately 28\% on
the test split of RefCOCO and RefCOCO+. Furthermore, GroundVLP performs
comparably to or even better than some non-VLP-based supervised models on the
Flickr30k entities dataset. Our code is available at
https://github.com/om-ai-lab/GroundVLP.