Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object
  Detection
Yifan Xu Mengdan Zhang Xiaoshan Yang Changsheng Xu
In this paper, we for the first time explore helpful multi-modal contextual
knowledge to understand novel categories for open-vocabulary object detection
(OVD). The multi-modal contextual knowledge stands for the joint relationship
across regions and words. However, it is challenging to incorporate such
multi-modal contextual knowledge into OVD. The reason is that previous
detection frameworks fail to jointly model multi-modal contextual knowledge, as
object detectors only support vision inputs and no caption description is
provided at test time. To this end, we propose a multi-modal contextual
knowledge distillation framework, MMC-Det, to transfer the learned contextual
knowledge from a teacher fusion transformer with diverse multi-modal masked
language modeling (D-MLM) to a student detector. The diverse multi-modal masked
language modeling is realized by an object divergence constraint upon
traditional multi-modal masked language modeling (MLM), in order to extract
fine-grained region-level visual contexts, which are vital to object detection.
Extensive experiments performed upon various detection datasets show the
effectiveness of our multi-modal context learning strategy, where our approach
well outperforms the recent state-of-the-art methods.