Augmenting Zero-Shot Detection Training with Image Labels
Katharina Kornmeier Ulla Scheler Pascal Herrmann
Zero-shot detection (ZSD), i.e., detection on classes not seen during
training, is essential for real world detection use-cases, but remains a
difficult task. Recent research attempts ZSD with detection models that output
embeddings instead of direct class labels. To this aim, the output of the
detection model must be aligned to a learned embedding space such as CLIP.
However, this alignment is hindered by detection data sets which are expensive
to produce compared to image classification annotations, and the resulting lack
of category diversity in the training data. We address this challenge by
leveraging the CLIP embedding space in combination with image labels from
ImageNet. Our results show that image labels are able to better align the
detector output to the embedding space and thus have a high potential for ZSD.
Compared to only training on detection data, we see a significant gain by
adding image label data of 3.3 mAP for the 65/15 split on COCO on the unseen
classes, i.e., we more than double the gain of related work.