Heron-Bench: A Benchmark for Evaluating Vision Language Models in
  Japanese
Yuichi Inoue Kento Sasaki Yuma Ochi Kazuki Fujii Kotaro Tanahashi Yu Yamaguchi
Vision Language Models (VLMs) have undergone a rapid evolution, giving rise
to significant advancements in the realm of multimodal understanding tasks.
However, the majority of these models are trained and evaluated on
English-centric datasets, leaving a gap in the development and evaluation of
VLMs for other languages, such as Japanese. This gap can be attributed to the
lack of methodologies for constructing VLMs and the absence of benchmarks to
accurately measure their performance. To address this issue, we introduce a
novel benchmark, Japanese Heron-Bench, for evaluating Japanese capabilities of
VLMs. The Japanese Heron-Bench consists of a variety of imagequestion answer
pairs tailored to the Japanese context. Additionally, we present a baseline
Japanese VLM that has been trained with Japanese visual instruction tuning
datasets. Our Heron-Bench reveals the strengths and limitations of the proposed
VLM across various ability dimensions. Furthermore, we clarify the capability
gap between strong closed models like GPT-4V and the baseline model, providing
valuable insights for future research in this domain. We release the benchmark
dataset and training code to facilitate further developments in Japanese VLM
research.