MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with
  Vision-Language Benchmark
Dongping Chen Ruoxi Chen Shilin Zhang Yinuo Liu Yaochen Wang Huichi Zhou Qihui Zhang Pan Zhou Yao Wan Lichao Sun
Multimodal Large Language Models (MLLMs) have gained significant attention
recently, showing remarkable potential in artificial general intelligence.
However, assessing the utility of MLLMs presents considerable challenges,
primarily due to the absence multimodal benchmarks that align with human
preferences. Inspired by LLM-as-a-Judge in LLMs, this paper introduces a novel
benchmark, termed MLLM-as-a-Judge, to assess the ability of MLLMs in assisting
judges including three distinct tasks: Scoring Evaluation, Pair Comparison, and
Batch Ranking. Our study reveals that, while MLLMs demonstrate remarkable
human-like discernment in Pair Comparisons, there is a significant divergence
from human preferences in Scoring Evaluation and Batch Ranking tasks.
Furthermore, MLLMs still face challenges in judgment, including diverse biases,
hallucinatory responses, and inconsistencies, even for advanced models such as
GPT-4V. These findings emphasize the pressing need for enhancements and further
research efforts regarding MLLMs as fully reliable evaluators. Code and dataset
are available at https://github.com/Dongping-Chen/MLLM-as-a-Judge.