ScaleDet: A Scalable Multi-Dataset Object Detector
Yanbei Chen Manchen Wang Abhay Mittal Zhenlin Xu Paolo Favaro Joseph Tighe Davide Modolo
Multi-dataset training provides a viable solution for exploiting
heterogeneous large-scale datasets without extra annotation cost. In this work,
we propose a scalable multi-dataset detector (ScaleDet) that can scale up its
generalization across datasets when increasing the number of training datasets.
Unlike existing multi-dataset learners that mostly rely on manual relabelling
efforts or sophisticated optimizations to unify labels across datasets, we
introduce a simple yet scalable formulation to derive a unified semantic label
space for multi-dataset training. ScaleDet is trained by visual-textual
alignment to learn the label assignment with label semantic similarities across
datasets. Once trained, ScaleDet can generalize well on any given upstream and
downstream datasets with seen and unseen classes. We conduct extensive
experiments using LVIS, COCO, Objects365, OpenImages as upstream datasets, and
13 datasets from Object Detection in the Wild (ODinW) as downstream datasets.
Our results show that ScaleDet achieves compelling strong model performance
with an mAP of 50.7 on LVIS, 58.8 on COCO, 46.8 on Objects365, 76.2 on
OpenImages, and 71.8 on ODinW, surpassing state-of-the-art detectors with the
same backbone.