Multi-Modal Mutual Attention and Iterative Interaction for Referring
  Image Segmentation
Chang Liu Henghui Ding Yulun Zhang Xudong Jiang
We address the problem of referring image segmentation that aims to generate
a mask for the object specified by a natural language expression. Many recent
works utilize Transformer to extract features for the target object by
aggregating the attended visual regions. However, the generic attention
mechanism in Transformer only uses the language input for attention weight
calculation, which does not explicitly fuse language features in its output.
Thus, its output feature is dominated by vision information, which limits the
model to comprehensively understand the multi-modal information, and brings
uncertainty for the subsequent mask decoder to extract the output mask. To
address this issue, we propose Multi-Modal Mutual Attention ($\mathrm{M^3Att}$)
and Multi-Modal Mutual Decoder ($\mathrm{M^3Dec}$) that better fuse information
from the two input modalities. Based on {$\mathrm{M^3Dec}$}, we further propose
Iterative Multi-modal Interaction ($\mathrm{IMI}$) to allow continuous and
in-depth interactions between language and vision features. Furthermore, we
introduce Language Feature Reconstruction ($\mathrm{LFR}$) to prevent the
language information from being lost or distorted in the extracted feature.
Extensive experiments show that our proposed approach significantly improves
the baseline and outperforms state-of-the-art referring image segmentation
methods on RefCOCO series datasets consistently.