Context Does Matter: End-to-end Panoptic Narrative Grounding with
  Deformable Attention Refined Matching Network
Yiming Lin Xiao-Bo Jin Qiufeng Wang Kaizhu Huang
Panoramic Narrative Grounding (PNG) is an emerging visual grounding task that
aims to segment visual objects in images based on dense narrative captions. The
current state-of-the-art methods first refine the representation of phrase by
aggregating the most similar $k$ image pixels, and then match the refined text
representations with the pixels of the image feature map to generate
segmentation results. However, simply aggregating sampled image features
ignores the contextual information, which can lead to phrase-to-pixel
mis-match. In this paper, we propose a novel learning framework called
Deformable Attention Refined Matching Network (DRMN), whose main idea is to
bring deformable attention in the iterative process of feature learning to
incorporate essential context information of different scales of pixels. DRMN
iteratively re-encodes pixels with the deformable attention network after
updating the feature representation of the top-$k$ most similar pixels. As
such, DRMN can lead to accurate yet discriminative pixel representations,
purify the top-$k$ most similar pixels, and consequently alleviate the
phrase-to-pixel mis-match substantially.Experimental results show that our
novel design significantly improves the matching results between text phrases
and image pixels. Concretely, DRMN achieves new state-of-the-art performance on
the PNG benchmark with an average recall improvement 3.5%. The codes are
available in: https://github.com/JaMesLiMers/DRMN.