Do Large Language Models Solve ARC Visual Analogies Like People Do?
Gustaw Opiełka Hannes Rosenbusch Veerle Vijverberg Claire E. Stevenson
The Abstraction Reasoning Corpus (ARC) is a visual analogical reasoning test
designed for humans and machines (Chollet, 2019). We compared human and large
language model (LLM) performance on a new child-friendly set of ARC items.
Results show that both children and adults outperform most LLMs on these tasks.
Error analysis revealed a similar "fallback" solution strategy in LLMs and
young children, where part of the analogy is simply copied. In addition, we
found two other error types, one based on seemingly grasping key concepts
(e.g., Inside-Outside) and the other based on simple combinations of analogy
input matrices. On the whole, "concept" errors were more common in humans, and
"matrix" errors were more common in LLMs. This study sheds new light on LLM
reasoning ability and the extent to which we can use error analyses and
comparisons with human development to understand how LLMs solve visual
analogies.