Referring Image Segmentation Using Text Supervision
Fang Liu Yuhao Liu Yuqiu Kong Ke Xu Lihe Zhang Baocai Yin Gerhard Hancke Rynson Lau
Existing Referring Image Segmentation (RIS) methods typically require
expensive pixel-level or box-level annotations for supervision. In this paper,
we observe that the referring texts used in RIS already provide sufficient
information to localize the target object. Hence, we propose a novel
weakly-supervised RIS framework to formulate the target localization problem as
a classification process to differentiate between positive and negative text
expressions. While the referring text expressions for an image are used as
positive expressions, the referring text expressions from other images can be
used as negative expressions for this image. Our framework has three main
novelties. First, we propose a bilateral prompt method to facilitate the
classification process, by harmonizing the domain discrepancy between visual
and linguistic features. Second, we propose a calibration method to reduce
noisy background information and improve the correctness of the response maps
for target object localization. Third, we propose a positive response map
selection strategy to generate high-quality pseudo-labels from the enhanced
response maps, for training a segmentation network for RIS inference. For
evaluation, we propose a new metric to measure localization accuracy.
Experiments on four benchmarks show that our framework achieves promising
performances to existing fully-supervised RIS methods while outperforming
state-of-the-art weakly-supervised methods adapted from related areas. Code is
available at https://github.com/fawnliu/TRIS.