CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection
Jiajin Tang Ge Zheng Jingyi Yu Sibei Yang
Task driven object detection aims to detect object instances suitable for
affording a task in an image. Its challenge lies in object categories available
for the task being too diverse to be limited to a closed set of object
vocabulary for traditional object detection. Simply mapping categories and
visual features of common objects to the task cannot address the challenge. In
this paper, we propose to explore fundamental affordances rather than object
categories, i.e., common attributes that enable different objects to accomplish
the same task. Moreover, we propose a novel multi-level chain-of-thought
prompting (MLCoT) to extract the affordance knowledge from large language
models, which contains multi-level reasoning steps from task to object examples
to essential visual attributes with rationales. Furthermore, to fully exploit
knowledge to benefit object recognition and localization, we propose a
knowledge-conditional detection framework, namely CoTDet. It conditions the
detector from the knowledge to generate object queries and regress boxes.
Experimental results demonstrate that our CoTDet outperforms state-of-the-art
methods consistently and significantly (+15.6 box AP and +14.8 mask AP) and can
generate rationales for why objects are detected to afford the task.