Lumen: Unleashing Versatile Vision-Centric Capabilities of Large
  Multimodal Models
Yang Jiao Shaoxiang Chen Zequn Jie Jingjing Chen Lin Ma Yu-Gang Jiang
Large Multimodal Model (LMM) is a hot research topic in the computer vision
area and has also demonstrated remarkable potential across multiple
disciplinary fields. A recent trend is to further extend and enhance the
perception capabilities of LMMs. The current methods follow the paradigm of
adapting the visual task outputs to the format of the language model, which is
the main component of a LMM. This adaptation leads to convenient development of
such LMMs with minimal modifications, however, it overlooks the intrinsic
characteristics of diverse visual tasks and hinders the learning of perception
capabilities. To address this issue, we propose a novel LMM architecture named
Lumen, a Large multimodal model with versatile vision-centric capability
enhancement. We decouple the LMM's learning of perception capabilities into
task-agnostic and task-specific stages. Lumen first promotes fine-grained
vision-language concept alignment, which is the fundamental capability for
various visual tasks. Thus the output of the task-agnostic stage is a shared
representation for all the tasks we address in this paper. Then the
task-specific decoding is carried out by flexibly routing the shared
representation to lightweight task decoders with negligible training efforts.
Benefiting from such a decoupled design, our Lumen surpasses existing LMM-based
approaches on the COCO detection benchmark with a clear margin and exhibits
seamless scalability to additional visual tasks. Furthermore, we also conduct
comprehensive ablation studies and generalization evaluations for deeper
insights. The code will be released at https://github.com/SxJyJay/Lumen.