ChEF: A Comprehensive Evaluation Framework for Standardized Assessment
  of Multimodal Large Language Models
Zhelun Shi Zhipin Wang Hongxing Fan Zhenfei Yin Lu Sheng Yu Qiao Jing Shao
Multimodal Large Language Models (MLLMs) have shown impressive abilities in
interacting with visual content with myriad potential downstream tasks.
However, even though a list of benchmarks has been proposed, the capabilities
and limitations of MLLMs are still not comprehensively understood, due to a
lack of a standardized and holistic evaluation framework. To this end, we
present the first Comprehensive Evaluation Framework (ChEF) that can
holistically profile each MLLM and fairly compare different MLLMs. First, we
structure ChEF as four modular components, i.e., Scenario as scalable
multimodal datasets, Instruction as flexible instruction retrieving formulae,
Inferencer as reliable question answering strategies, and Metric as indicative
task-specific score functions. Based on them, ChEF facilitates versatile
evaluations in a standardized framework, and new evaluations can be built by
designing new Recipes (systematic selection of these four components). Notably,
current MLLM benchmarks can be readily summarized as recipes of ChEF. Second,
we introduce 6 new recipes to quantify competent MLLMs' desired capabilities
(or called desiderata, i.e., calibration, in-context learning, instruction
following, language performance, hallucination, and robustness) as reliable
agents that can perform real-world multimodal interactions. Third, we conduct a
large-scale evaluation of 9 prominent MLLMs on 9 scenarios and 6 desiderata.
Our evaluation summarized over 20 valuable observations concerning the
generalizability of MLLMs across various scenarios and the composite capability
of MLLMs required for multimodal interactions. We will publicly release all the
detailed implementations for further analysis, as well as an easy-to-use
modular toolkit for the integration of new recipes and models, so that ChEF can
be a growing evaluation framework for the MLLM community.