Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained
  Evaluation
Seongyun Lee Seungone Kim Sue Hyun Park Geewook Kim Minjoon Seo
Assessing long-form responses generated by Vision-Language Models (VLMs) is
challenging. It not only requires checking whether the VLM follows the given
instruction but also verifying whether the text output is properly grounded on
the given image. Inspired by the recent approach of evaluating LMs with LMs, in
this work, we propose to evaluate VLMs with VLMs. For this purpose, we present
a new feedback dataset called the Perception Collection, encompassing 15K
customized score rubrics that users might care about during assessment. Using
the Perception Collection, we train Prometheus-Vision, the first open-source
VLM evaluator model that can understand the user-defined score criteria during
evaluation. Prometheus-Vision shows the highest Pearson correlation with human
evaluators and GPT-4V among open-source models, showing its effectiveness for
transparent and accessible evaluation of VLMs. We open-source our code,
dataset, and model at https://github.com/kaistAI/prometheus-vision