Open World Object Detection in the Era of Foundation Models
Orr Zohar Alejandro Lozano Shelly Goel Serena Yeung Kuan-Chieh Wang
Object detection is integral to a bevy of real-world applications, from
robotics to medical image analysis. To be used reliably in such applications,
models must be capable of handling unexpected - or novel - objects. The open
world object detection (OWD) paradigm addresses this challenge by enabling
models to detect unknown objects and learn discovered ones incrementally.
However, OWD method development is hindered due to the stringent benchmark and
task definitions. These definitions effectively prohibit foundation models.
Here, we aim to relax these definitions and investigate the utilization of
pre-trained foundation models in OWD. First, we show that existing benchmarks
are insufficient in evaluating methods that utilize foundation models, as even
naive integration methods nearly saturate these benchmarks. This result
motivated us to curate a new and challenging benchmark for these models.
Therefore, we introduce a new benchmark that includes five real-world
application-driven datasets, including challenging domains such as aerial and
surgical images, and establish baselines. We exploit the inherent connection
between classes in application-driven datasets and introduce a novel method,
Foundation Object detection Model for the Open world, or FOMO, which identifies
unknown objects based on their shared attributes with the base known objects.
FOMO has ~3x unknown object mAP compared to baselines on our benchmark.
However, our results indicate a significant place for improvement - suggesting
a great research opportunity in further scaling object detection methods to
real-world domains. Our code and benchmark are available at
https://orrzohar.github.io/projects/fomo/.