Detection-Oriented Image-Text Pretraining for Open-Vocabulary Detection
Dahun Kim Anelia Angelova Weicheng Kuo
We present a new open-vocabulary detection approach based on
detection-oriented image-text pretraining to bridge the gap between image-level
pretraining and open-vocabulary object detection. At the pretraining phase, we
replace the commonly used classification architecture with the detector
architecture, which better serves the region-level recognition needs of
detection by enabling the detector heads to learn from noisy image-text pairs.
Using only standard contrastive loss and no pseudo-labeling, our approach is a
simple yet effective extension of the contrastive learning method to learn
emergent object-semantic cues. In addition, we propose a shifted-window
learning approach upon window attention to make the backbone representation
more robust, translation-invariant, and less biased by the window pattern. On
the popular LVIS open-vocabulary detection benchmark, our approach sets a new
state of the art of 40.4 mask AP$_r$ using the common ViT-L backbone,
significantly outperforming the best existing approach by +6.5 mask AP$_r$ at
system level. On the COCO benchmark, we achieve very competitive 40.8 novel AP
without pseudo labeling or weak supervision. In addition, we evaluate our
approach on the transfer detection setup, where ours outperforms the baseline
significantly. Visualization reveals emerging object locality from the
pretraining recipes compared to the baseline. Code and models will be publicly
released.