GenCeption: Evaluate Multimodal LLMs with Unlabeled Unimodal Data
Lele Cao Valentin Buchner Zineb Senane Fangkai Yang
Multimodal Large Language Models (MLLMs) are commonly evaluated using costly
annotated multimodal benchmarks. However, these benchmarks often struggle to
keep pace with the rapidly advancing requirements of MLLM evaluation. We
propose GenCeption, a novel and annotation-free MLLM evaluation framework that
merely requires unimodal data to assess inter-modality semantic coherence and
inversely reflects the models' inclination to hallucinate. Analogous to the
popular DrawCeption game, GenCeption initiates with a non-textual sample and
undergoes a series of iterative description and generation steps. Semantic
drift across iterations is quantified using the GC@T metric. Our empirical
findings validate GenCeption's efficacy, showing strong correlations with
popular MLLM benchmarking results. GenCeption may be extended to mitigate
training data contamination by utilizing ubiquitous, previously unseen unimodal
data.