Large Multimodal Agents: A Survey
Junlin Xie Zhihong Chen Ruifei Zhang Xiang Wan Guanbin Li
Large language models (LLMs) have achieved superior performance in powering
text-based AI agents, endowing them with decision-making and reasoning
abilities akin to humans. Concurrently, there is an emerging research trend
focused on extending these LLM-powered AI agents into the multimodal domain.
This extension enables AI agents to interpret and respond to diverse multimodal
user queries, thereby handling more intricate and nuanced tasks. In this paper,
we conduct a systematic review of LLM-driven multimodal agents, which we refer
to as large multimodal agents ( LMAs for short). First, we introduce the
essential components involved in developing LMAs and categorize the current
body of research into four distinct types. Subsequently, we review the
collaborative frameworks integrating multiple LMAs , enhancing collective
efficacy. One of the critical challenges in this field is the diverse
evaluation methods used across existing studies, hindering effective comparison
among different LMAs . Therefore, we compile these evaluation methodologies and
establish a comprehensive framework to bridge the gaps. This framework aims to
standardize evaluations, facilitating more meaningful comparisons. Concluding
our review, we highlight the extensive applications of LMAs and propose
possible future research directions. Our discussion aims to provide valuable
insights and guidelines for future research in this rapidly evolving field. An
up-to-date resource list is available at
https://github.com/jun0wanan/awesome-large-multimodal-agents.