Semi-Supervised Panoptic Narrative Grounding
Danni Yang Jiayi Ji Xiaoshuai Sun Haowei Wang Yinan Li Yiwei Ma Rongrong Ji
Despite considerable progress, the advancement of Panoptic Narrative
Grounding (PNG) remains hindered by costly annotations. In this paper, we
introduce a novel Semi-Supervised Panoptic Narrative Grounding (SS-PNG)
learning scheme, capitalizing on a smaller set of labeled image-text pairs and
a larger set of unlabeled pairs to achieve competitive performance. Unlike
visual segmentation tasks, PNG involves one pixel belonging to multiple
open-ended nouns. As a result, existing multi-class based semi-supervised
segmentation frameworks cannot be directly applied to this task. To address
this challenge, we first develop a novel SS-PNG Network (SS-PNG-NW) tailored to
the SS-PNG setting. We thoroughly investigate strategies such as Burn-In and
data augmentation to determine the optimal generic configuration for the
SS-PNG-NW. Additionally, to tackle the issue of imbalanced pseudo-label
quality, we propose a Quality-Based Loss Adjustment (QLA) approach to adjust
the semi-supervised objective, resulting in an enhanced SS-PNG-NW+. Employing
our proposed QLA, we improve BCE Loss and Dice loss at pixel and mask levels,
respectively. We conduct extensive experiments on PNG datasets, with our
SS-PNG-NW+ demonstrating promising results comparable to fully-supervised
models across all data ratios. Remarkably, our SS-PNG-NW+ outperforms
fully-supervised models with only 30% and 50% supervision data, exceeding their
performance by 0.8% and 1.1% respectively. This highlights the effectiveness of
our proposed SS-PNG-NW+ in overcoming the challenges posed by limited
annotations and enhancing the applicability of PNG tasks. The source code is
available at https://github.com/nini0919/SSPNG.