DesCo: Learning Object Recognition with Rich Language Descriptions
Liunian Harold Li Zi-Yi Dou Nanyun Peng Kai-Wei Chang
Recent development in vision-language approaches has instigated a paradigm
shift in learning visual recognition models from language supervision. These
approaches align objects with language queries (e.g. "a photo of a cat") and
improve the models' adaptability to identify novel objects and domains.
Recently, several studies have attempted to query these models with complex
language expressions that include specifications of fine-grained semantic
details, such as attributes, shapes, textures, and relations. However, simply
incorporating language descriptions as queries does not guarantee accurate
interpretation by the models. In fact, our experiments show that GLIP, the
state-of-the-art vision-language model for object detection, often disregards
contextual information in the language descriptions and instead relies heavily
on detecting objects solely by their names. To tackle the challenges, we
propose a new description-conditioned (DesCo) paradigm of learning object
recognition models with rich language descriptions consisting of two major
innovations: 1) we employ a large language model as a commonsense knowledge
engine to generate rich language descriptions of objects based on object names
and the raw image-text caption; 2) we design context-sensitive queries to
improve the model's ability in deciphering intricate nuances embedded within
descriptions and enforce the model to focus on context rather than object names
alone. On two novel object detection benchmarks, LVIS and OminiLabel, under the
zero-shot detection setting, our approach achieves 34.8 APr minival (+9.1) and
29.3 AP (+3.6), respectively, surpassing the prior state-of-the-art models,
GLIP and FIBER, by a large margin.