Revisiting Few-Shot Object Detection with Vision-Language Models
Anish Madan Neehar Peri Shu Kong Deva Ramanan
Few-shot object detection (FSOD) benchmarks have advanced techniques for
detecting new categories with limited annotations. Existing benchmarks
repurpose well-established datasets like COCO by partitioning categories into
base and novel classes for pre-training and fine-tuning respectively. However,
these benchmarks do not reflect how FSOD is deployed in practice. Rather than
only pre-training on a small number of base categories, we argue that it is
more practical to fine-tune a foundation model (e.g., a vision-language model
(VLM) pre-trained on web-scale data) for a target domain. Surprisingly, we find
that zero-shot inference from VLMs like GroundingDINO significantly outperforms
the state-of-the-art (48.3 vs. 33.1 AP) on COCO. However, such zero-shot models
can still be misaligned to target concepts of interest. For example, trailers
on the web may be different from trailers in the context of autonomous
vehicles. In this work, we propose Foundational FSOD, a new benchmark protocol
that evaluates detectors pre-trained on any external datasets and fine-tuned on
K-shots per target class. Further, we note that current FSOD benchmarks are
actually federated datasets containing exhaustive annotations for each category
on a subset of the data. We leverage this insight to propose simple strategies
for fine-tuning VLMs with federated losses. We demonstrate the effectiveness of
our approach on LVIS and nuImages, improving over prior work by 5.9 AP.