OlympiadBench: A Challenging Benchmark for Promoting AGI with
  Olympiad-Level Bilingual Multimodal Scientific Problems
Chaoqun He Renjie Luo Yuzhuo Bai Shengding Hu Zhen Leng Thai Junhao Shen Jinyi Hu Xu Han Yujie Huang Yuxiang Zhang Jie Liu Lei Qi Zhiyuan Liu Maosong Sun
Recent advancements have seen Large Language Models (LLMs) and Large
Multimodal Models (LMMs) surpassing general human capabilities in various
tasks, approaching the proficiency level of human experts across multiple
domains. With traditional benchmarks becoming less challenging for these
models, new rigorous challenges are essential to gauge their advanced
abilities. In this work, we present OlympiadBench, an Olympiad-level bilingual
multimodal scientific benchmark, featuring 8,952 problems from Olympiad-level
mathematics and physics competitions, including the Chinese college entrance
exam. Each problem is detailed with expert-level annotations for step-by-step
reasoning. Evaluating top-tier models on OlympiadBench, we implement a
comprehensive assessment methodology to accurately evaluate model responses.
Notably, the best-performing model, GPT-4V, attains an average score of 17.23%
on OlympiadBench, with a mere 11.28% in physics, highlighting the benchmark
rigor and the intricacy of physical reasoning. Our analysis orienting GPT-4V
points out prevalent issues with hallucinations, knowledge omissions, and
logical fallacies. We hope that our challenging benchmark can serve as a
valuable resource for helping future AGI research endeavors.