Enhancing Subtask Performance of Multi-modal Large Language Model
Yongqiang Zhao Zhenyu Li Feng Zhang Xinhai Xu Donghong Liu
Multi-modal Large Language Model (MLLM) refers to a model expanded from a
Large Language Model (LLM) that possesses the capability to handle and infer
multi-modal data. Current MLLMs typically begin by using LLMs to decompose
tasks into multiple subtasks, then employing individual pre-trained models to
complete specific subtasks, and ultimately utilizing LLMs to integrate the
results of each subtasks to obtain the results of the task. In real-world
scenarios, when dealing with large projects, it is common practice to break
down the project into smaller sub-projects, with different teams providing
corresponding solutions or results. The project owner then decides which
solution or result to use, ensuring the best possible outcome for each subtask
and, consequently, for the entire project. Inspired by this, this study
considers selecting multiple pre-trained models to complete the same subtask.
By combining the results from multiple pre-trained models, the optimal subtask
result is obtained, enhancing the performance of the MLLM. Specifically, this
study first selects multiple pre-trained models focused on the same subtask
based on distinct evaluation approaches, and then invokes these models in
parallel to process input data and generate corresponding subtask results.
Finally, the results from multiple pre-trained models for the same subtask are
compared using the LLM, and the best result is chosen as the outcome for that
subtask. Extensive experiments are conducted in this study using GPT-4
annotated datasets and human-annotated datasets. The results of various
evaluation metrics adequately demonstrate the effectiveness of the proposed
approach in this paper.