Generalized Category Discovery in Semantic Segmentation
Zhengyuan Peng Qijian Tian Jianqing Xu Yizhang Jin Xuequan Lu Xin Tan Yuan Xie Lizhuang Ma
This paper explores a novel setting called Generalized Category Discovery in
Semantic Segmentation (GCDSS), aiming to segment unlabeled images given prior
knowledge from a labeled set of base classes. The unlabeled images contain
pixels of the base class or novel class. In contrast to Novel Category
Discovery in Semantic Segmentation (NCDSS), there is no prerequisite for prior
knowledge mandating the existence of at least one novel class in each unlabeled
image. Besides, we broaden the segmentation scope beyond foreground objects to
include the entire image. Existing NCDSS methods rely on the aforementioned
priors, making them challenging to truly apply in real-world situations. We
propose a straightforward yet effective framework that reinterprets the GCDSS
challenge as a task of mask classification. Additionally, we construct a
baseline method and introduce the Neighborhood Relations-Guided Mask Clustering
Algorithm (NeRG-MaskCA) for mask categorization to address the fragmentation in
semantic representation. A benchmark dataset, Cityscapes-GCD, derived from the
Cityscapes dataset, is established to evaluate the GCDSS framework. Our method
demonstrates the feasibility of the GCDSS problem and the potential for
discovering and segmenting novel object classes in unlabeled images. We employ
the generated pseudo-labels from our approach as ground truth to supervise the
training of other models, thereby enabling them with the ability to segment
novel classes. It paves the way for further research in generalized category
discovery, broadening the horizons of semantic segmentation and its
applications. For details, please visit https://github.com/JethroPeng/GCDSS