Multi-Modal Prototypes for Open-Set Semantic Segmentation
Yuhuan Yang Chaofan Ma Chen Ju Ya Zhang Yanfeng Wang
In semantic segmentation, adapting a visual system to novel object categories
at inference time has always been both valuable and challenging. To enable such
generalization, existing methods rely on either providing several support
examples as visual cues or class names as textual cues. Through the development
is relatively optimistic, these two lines have been studied in isolation,
neglecting the complementary intrinsic of low-level visual and high-level
language information. In this paper, we define a unified setting termed as
open-set semantic segmentation (O3S), which aims to learn seen and unseen
semantics from both visual examples and textual names. Our pipeline extracts
multi-modal prototypes for segmentation task, by first single modal
self-enhancement and aggregation, then multi-modal complementary fusion. To be
specific, we aggregate visual features into several tokens as visual
prototypes, and enhance the class name with detailed descriptions for textual
prototype generation. The two modalities are then fused to generate multi-modal
prototypes for final segmentation. On both \pascal and \coco datasets, we
conduct extensive experiments to evaluate the framework effectiveness.
State-of-the-art results are achieved even on more detailed part-segmentation,
Pascal-Animals, by only training on coarse-grained datasets. Thorough ablation
studies are performed to dissect each component, both quantitatively and
qualitatively.