DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model
Xiuye Gu Yin Cui Jonathan Huang Abdullah Rashwan Xuan Yang Xingyi Zhou Golnaz Ghiasi Weicheng Kuo Huizhong Chen Liang-Chieh Chen David A Ross
Observing the close relationship among panoptic, semantic and instance
segmentation tasks, we propose to train a universal multi-dataset multi-task
segmentation model: DaTaSeg.We use a shared representation (mask proposals with
class predictions) for all tasks. To tackle task discrepancy, we adopt
different merge operations and post-processing for different tasks. We also
leverage weak-supervision, allowing our segmentation model to benefit from
cheaper bounding box annotations. To share knowledge across datasets, we use
text embeddings from the same semantic embedding space as classifiers and share
all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO
panoptic, and Objects365 detection datasets. DaTaSeg improves performance on
all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE
semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised
knowledge transfer on ADE panoptic and Objects365 instance segmentation.
Experiments show DaTaSeg scales with the number of training datasets and
enables open-vocabulary segmentation through direct transfer. In addition, we
annotate an Objects365 instance segmentation set of 1,000 images and will
release it as a public benchmark.