Visual Question Answering Instruction: Unlocking Multimodal Large
  Language Model To Domain-Specific Visual Multitasks
Jusung Lee Sungguk Cha Younghyun Lee Cheoljong Yang
Having revolutionized natural language processing (NLP) applications, large
language models (LLMs) are expanding into the realm of multimodal inputs. Owing
to their ability to interpret images, multimodal LLMs (MLLMs) have been
primarily used for vision-language tasks. Currently, MLLMs have not yet been
extended for domain-specific visual tasks, which require a more explicit
understanding of visual information. We developed a method to transform
domain-specific visual and vision-language datasets into a unified question
answering format called Visual Question Answering Instruction (VQA-IN), thereby
extending MLLM to domain-specific tasks. The VQA-IN was applied to train
multiple MLLM architectures using smaller versions of LLMs (sLLMs). The
experimental results indicated that the proposed method achieved a high score
metric on domainspecific visual tasks while also maintaining its performance on
vision-language tasks in a multitask manner.