Calibration & Reconstruction: Deep Integrated Language for Referring
  Image Segmentation
Yichen Yan Xingjian He Sihan Chen Jing Liu
Referring image segmentation aims to segment an object referred to by natural
language expression from an image. The primary challenge lies in the efficient
propagation of fine-grained semantic information from textual features to
visual features. Many recent works utilize a Transformer to address this
challenge. However, conventional transformer decoders can distort linguistic
information with deeper layers, leading to suboptimal results. In this paper,
we introduce CRFormer, a model that iteratively calibrates multi-modal features
in the transformer decoder. We start by generating language queries using
vision features, emphasizing different aspects of the input language. Then, we
propose a novel Calibration Decoder (CDec) wherein the multi-modal features can
iteratively calibrated by the input language features. In the Calibration
Decoder, we use the output of each decoder layer and the original language
features to generate new queries for continuous calibration, which gradually
updates the language features. Based on CDec, we introduce a Language
Reconstruction Module and a reconstruction loss. This module leverages queries
from the final layer of the decoder to reconstruct the input language and
compute the reconstruction loss. This can further prevent the language
information from being lost or distorted. Our experiments consistently show the
superior performance of our approach across RefCOCO, RefCOCO+, and G-Ref
datasets compared to state-of-the-art methods.