Lenna: Language Enhanced Reasoning Detection Assistant
Fei Wei Xinyu Zhang Ailing Zhang Bo Zhang Xiangxiang Chu
With the fast-paced development of multimodal large language models (MLLMs),
we can now converse with AI systems in natural languages to understand images.
However, the reasoning power and world knowledge embedded in the large language
models have been much less investigated and exploited for image perception
tasks. In this paper, we propose Lenna, a language-enhanced reasoning detection
assistant, which utilizes the robust multimodal feature representation of
MLLMs, while preserving location information for detection. This is achieved by
incorporating an additional <DET> token in the MLLM vocabulary that is free of
explicit semantic context but serves as a prompt for the detector to identify
the corresponding position. To evaluate the reasoning capability of Lenna, we
construct a ReasonDet dataset to measure its performance on reasoning-based
detection. Remarkably, Lenna demonstrates outstanding performance on ReasonDet
and comes with significantly low training costs. It also incurs minimal
transferring overhead when extended to other tasks. Our code and model will be
available at https://git.io/Lenna.