REBUS: A Robust Evaluation Benchmark of Understanding Symbols
Andrew Gritsevskiy Arjun Panickssery Aaron Kirtland Derik Kauffman Hans Gundlach Irina Gritsevskaya Joe Cavanagh Jonathan Chiang Lydia La Roux Michelle Hung
We propose a new benchmark evaluating the performance of multimodal large
language models on rebus puzzles. The dataset covers 333 original examples of
image-based wordplay, cluing 13 categories such as movies, composers, major
cities, and food. To achieve good performance on the benchmark of identifying
the clued word or phrase, models must combine image recognition and string
manipulation with hypothesis testing, multi-step reasoning, and an
understanding of human cognition, making for a complex, multimodal evaluation
of capabilities. We find that proprietary models such as GPT-4V and Gemini Pro
significantly outperform all other tested models. However, even the best model
has a final accuracy of just 24%, highlighting the need for substantial
improvements in reasoning. Further, models rarely understand all parts of a
puzzle, and are almost always incapable of retroactively explaining the correct
answer. Our benchmark can therefore be used to identify major shortcomings in
the knowledge and reasoning of multimodal large language models.